Sep 15 11:25:04.816810 ip-10-0-142-54 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 11:25:05.177323 ip-10-0-142-54 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:25:05.178109 ip-10-0-142-54 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 11:25:05.178109 ip-10-0-142-54 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:25:05.178109 ip-10-0-142-54 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 11:25:05.178109 ip-10-0-142-54 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 11:25:05.179740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.179625    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 11:25:05.182881 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182863    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:05.182881 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182880    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:05.182881 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182884    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182887    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182892    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182896    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182899    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182902    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182905    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182908    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182910    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182913    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182916    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182919    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182921    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182924    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182927    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182929    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182932    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182935    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182938    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182940    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:05.182990 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182943    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182946    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182949    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182951    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182954    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182957    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182960    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182970    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182973    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182976    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182978    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182981    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182984    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182986    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182990    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182993    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182996    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.182999    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183002    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183005    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:05.183516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183008    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183011    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183013    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183016    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183018    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183021    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183024    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183027    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183029    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183032    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183036    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183038    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183041    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183043    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183046    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183049    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183051    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183054    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183057    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183059    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:05.184083 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183062    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183066    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183070    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183073    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183076    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183079    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183084    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183087    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183090    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183092    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183096    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183099    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183101    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183104    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183107    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183110    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183115    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183118    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183122    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:05.184584 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183125    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183129    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183133    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183136    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183139    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183141    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183144    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183146    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183149    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183151    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183154    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183156    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183159    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183162    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183164    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183167    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183169    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183172    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183892    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183899    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:05.185056 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183902    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183905    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183909    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183911    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183914    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183916    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183919    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183922    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183925    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183928    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183930    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183933    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183936    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183939    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183942    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183945    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183947    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183950    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183952    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183955    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:05.185547 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183958    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183961    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183963    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183966    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183969    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183971    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183974    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183976    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183980    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183982    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183985    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183989    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183991    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183994    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.183998    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184000    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184003    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184005    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184010    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:05.186055 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184014    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184017    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184020    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184023    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184026    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184029    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184036    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184039    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184042    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184045    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184047    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184050    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184052    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184055    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184057    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184061    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184063    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184066    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184068    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184072    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:05.186541 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184075    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184078    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184081    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184084    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184088    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184091    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184095    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184098    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184101    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184103    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184106    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184109    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184111    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184114    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184117    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184119    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184122    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184124    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184127    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:05.187036 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184130    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184132    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184135    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184138    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184140    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184142    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184145    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184147    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184150    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184153    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184155    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184158    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184160    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184164    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184167    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184170    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184173    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184175    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184178    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184625    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:05.187519 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184631    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184635    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184638    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184641    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184644    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184647    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184650    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184652    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184655    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184658    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184661    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184663    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184667    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184669    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184672    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184675    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184677    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184680    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184682    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184685    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:05.188009 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184688    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184690    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184693    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184695    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184698    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184701    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184706    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184709    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184711    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184714    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184717    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184720    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184722    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184725    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184727    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184730    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184732    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184736    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184740    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:05.188533 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184743    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184746    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184749    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184752    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184755    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184757    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184760    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184763    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184766    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184768    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184772    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184774    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184777    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184779    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184782    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184784    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184788    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184792    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184794    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:05.188999 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184799    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184802    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184805    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184808    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184811    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184815    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184819    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184822    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184824    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184827    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184830    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184832    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184835    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184838    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184841    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184843    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184846    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184848    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184851    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:05.189479 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184854    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184857    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184860    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184863    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184865    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184868    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184870    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184873    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184875    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184878    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184880    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184883    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184885    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184890    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184893    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184896    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184898    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184901    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184904    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184906    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.184909    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:05.189989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.184979    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185015    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185022    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185027    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185032    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185036    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185040    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185045    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185048    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185052    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185056    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185059    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185063    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185067    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185071    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185074    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185077    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185080    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185084    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185087    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185090    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185093    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185097    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185100    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 11:25:05.190539 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185105    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185108    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185112    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185115    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185118    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185121    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185124    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185128    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185132    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185135    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185138    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185141    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185144    2574 flags.go:64] FLAG: --enable-server="true"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185147    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185152    2574 flags.go:64] FLAG: --event-burst="100"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185155    2574 flags.go:64] FLAG: --event-qps="50"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185158    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185161    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185164    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185168    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185171    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185174    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185178    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185181    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185184    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 11:25:05.191110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185187    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185190    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185192    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185195    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185198    2574 flags.go:64] FLAG: --feature-gates=""
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185202    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185205    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185209    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185212    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185215    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185218    2574 flags.go:64] FLAG: --help="false"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185221    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185225    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185227    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185231    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185234    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185238    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185241    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185244    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185246    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185249    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185252    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185255    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185258    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 15 11:25:05.191783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185261    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185264    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185268    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185271    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185274    2574 flags.go:64] FLAG: --lock-file=""
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185277    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185280    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185283    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185288    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185291    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185294    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185297    2574 flags.go:64] FLAG: --logging-format="text"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185299    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185303    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185306    2574 flags.go:64] FLAG: --manifest-url=""
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185309    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185314    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185317    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185322    2574 flags.go:64] FLAG: --max-pods="110"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185325    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185328    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185331    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185334    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185337    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185340    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 11:25:05.192408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185343    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185351    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185354    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185357    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185360    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185363    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185369    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185372    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185376    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185379    2574 flags.go:64] FLAG: --port="10250"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185382    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185385    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d0a903fcc5b38a2"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185388    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185391    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185395    2574 flags.go:64] FLAG: --register-node="true"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185398    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185402    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185405    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185408    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185410    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185414    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185418    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185420    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185425    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185429    2574 flags.go:64] FLAG: --runonce="false"
Sep 15 11:25:05.193048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185432    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185435    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185438    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185441    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185458    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185462    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185466    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185469    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185472    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185475    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185478    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185481    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185484    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185487    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185490    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185496    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185499    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185502    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185506    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185509    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185511    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185514    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185518    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185521    2574 flags.go:64] FLAG: --v="2"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185526    2574 flags.go:64] FLAG: --version="false"
Sep 15 11:25:05.193698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185530    2574 flags.go:64] FLAG: --vmodule=""
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185533    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185537    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185653    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185657    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185661    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185666    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185669    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185672    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185675    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185678    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185680    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185683    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185686    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185690    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185694    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185697    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185700    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185703    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:05.194322 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185706    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185709    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185712    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185715    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185718    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185720    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185723    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185726    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185729    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185732    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185735    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185738    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185741    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185744    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185747    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185750    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185752    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185755    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185758    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185762    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185765    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:05.194849 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185768    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185771    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185773    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185777    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185780    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185783    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185785    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185788    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185791    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185793    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185796    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185798    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185801    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185804    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185806    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185816    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185820    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185824    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185827    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:05.195363 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185830    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185833    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185836    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185838    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185842    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185845    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185847    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185850    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185853    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185855    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185860    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185864    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185866    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185869    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185872    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185874    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185877    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185880    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185883    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185885    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:05.195857 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185888    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185891    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185893    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185896    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185899    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185901    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185904    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185907    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185909    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185912    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185914    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185917    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185920    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185922    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185925    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185927    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185930    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185933    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185935    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185938    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:05.196349 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185940    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185943    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.185946    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.185953    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.193711    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.193727    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.194483    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.194494    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194555    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194560    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194564    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194567    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194570    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194573    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:05.196858 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194576    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194581    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194585    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194589    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194592    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194595    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194597    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194600    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194604    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194606    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194610    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194613    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194615    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194618    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194621    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194624    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194626    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194629    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194632    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194635    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:05.197219 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194637    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194640    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194643    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194646    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194649    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194652    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194655    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194657    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194660    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194663    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194665    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194668    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194670    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194673    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194675    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194678    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194681    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194684    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194686    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194690    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:05.197776 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194693    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194696    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194699    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194702    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194705    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194707    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194710    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194712    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194716    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194719    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194722    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194725    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194727    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194730    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194732    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194736    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194738    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194741    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194744    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:05.198316 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194746    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194749    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194751    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194754    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194757    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194759    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194762    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194764    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194767    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194769    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194772    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194774    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194777    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194779    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194782    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194784    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194787    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194789    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194792    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194795    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:05.198796 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194797    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194800    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194803    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194808    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194810    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194814    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194817    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194821    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194825    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194828    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194830    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194833    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194836    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194838    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.194843    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:25:05.199270 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194957    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194962    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194965    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194968    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194971    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194974    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194976    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194979    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194982    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194984    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194987    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194989    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194992    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194995    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.194997    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195000    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195003    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195007    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195010    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195013    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 11:25:05.199673 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195016    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195019    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195022    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195025    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195028    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195031    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195034    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195036    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195040    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195044    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195047    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195050    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195053    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195055    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195058    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195060    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195063    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195065    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195068    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195070    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 11:25:05.200158 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195073    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195076    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195080    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195083    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195086    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195088    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195091    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195093    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195096    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195098    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195101    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195103    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195106    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195109    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195111    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195114    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195118    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195121    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195124    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 11:25:05.200687 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195127    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195130    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195132    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195135    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195138    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195140    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195143    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195145    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195148    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195150    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195153    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195155    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195158    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195160    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195163    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195166    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195168    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195171    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195173    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195176    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 11:25:05.201157 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195178    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195181    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195183    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195186    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195188    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195191    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195193    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195196    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195199    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195202    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195205    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195208    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195210    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195213    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195215    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195218    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195221    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195223    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195226    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 11:25:05.201689 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:05.195229    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.195234    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.195354    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.197334    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.198110    2574 server.go:1019] "Starting client certificate rotation"
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.198207    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.198242    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 11:25:05.202154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.200497    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 11:25:05.202334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.202318    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 11:25:05.209951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.209931    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 15 11:25:05.213768 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.213753    2574 log.go:25] "Validated CRI v1 image API"
Sep 15 11:25:05.214781 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.214765    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 11:25:05.220603 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.220576    2574 fs.go:135] Filesystem UUIDs: map[31813dc3-e5f4-4e9e-a542-90f803927a4c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cbeb66a8-ef49-4484-b7eb-34c3daaad73b:/dev/nvme0n1p4]
Sep 15 11:25:05.220674 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.220601    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 11:25:05.223565 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.223548    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 11:25:05.226391 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.226283    2574 manager.go:216] Machine: {Timestamp:2026-09-15 11:25:05.224127172 +0000 UTC m=+0.299852620 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103157 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0d8726a901407c5803cd12b7c764 SystemUUID:ec2e0d87-26a9-0140-7c58-03cd12b7c764 BootID:b418fcac-8164-4148-8768-7ac79199999b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d6:95:f7:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d6:95:f7:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:63:6a:ce:29:c5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 11:25:05.226391 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.226390    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 11:25:05.226531 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.226519    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 11:25:05.229060 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.229038    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 11:25:05.229193 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.229062    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-54.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 11:25:05.229686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.229676    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 11:25:05.229727 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.229689    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 11:25:05.229727 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.229697    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 11:25:05.230229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.230219    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 11:25:05.230918 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.230909    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 11:25:05.233576 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233565    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 11:25:05.233692 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233682    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 11:25:05.233728 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233711    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 11:25:05.233728 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233723    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 11:25:05.233783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233735    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 11:25:05.233783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233744    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 15 11:25:05.233783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.233758    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 11:25:05.234142 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.234129    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 11:25:05.234187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.234148    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 11:25:05.236042 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.236028    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 11:25:05.238109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.238096    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 11:25:05.238154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.238126    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 11:25:05.239507 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239466    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 11:25:05.239541 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239516    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 11:25:05.239541 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239524    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 11:25:05.239541 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239534    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239543    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239549    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239563    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239568    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239575    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239580    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239595    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 11:25:05.239624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.239601    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 11:25:05.240348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.240340    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 11:25:05.240348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.240349    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 11:25:05.244262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.244237    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-54.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 11:25:05.244753 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.244718    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 11:25:05.244870 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.244818    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-54.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 11:25:05.245833 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.245816    2574 server.go:1268] "Started kubelet"
Sep 15 11:25:05.245833 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.245822    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8rz7s"
Sep 15 11:25:05.245965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.245871    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 11:25:05.245965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.245909    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 11:25:05.246035 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.245984    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 11:25:05.246754 ip-10-0-142-54 systemd[1]: Started Kubernetes Kubelet.
Sep 15 11:25:05.247432 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.247415    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 11:25:05.247787 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.247774    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 11:25:05.250940 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.250891    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 11:25:05.251291 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.249540    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-54.ec2.internal.18d57ada5ddbaae4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-54.ec2.internal,UID:ip-10-0-142-54.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-54.ec2.internal,},FirstTimestamp:2026-09-15 11:25:05.245776612 +0000 UTC m=+0.321502059,LastTimestamp:2026-09-15 11:25:05.245776612 +0000 UTC m=+0.321502059,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-54.ec2.internal,}"
Sep 15 11:25:05.252827 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.252811    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 11:25:05.253216 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.253198    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8rz7s"
Sep 15 11:25:05.253337 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.253325    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 11:25:05.253485 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.253466    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.253959 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.253933    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 11:25:05.254043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.253930    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 11:25:05.254043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.253995    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 11:25:05.254145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254098    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 11:25:05.254145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254107    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 11:25:05.254145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254129    2574 factory.go:55] Registering systemd factory
Sep 15 11:25:05.254145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254146    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 15 11:25:05.254394 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254377    2574 factory.go:153] Registering CRI-O factory
Sep 15 11:25:05.254485 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254396    2574 factory.go:223] Registration of the crio container factory successfully
Sep 15 11:25:05.254547 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254498    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 11:25:05.254547 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254521    2574 factory.go:103] Registering Raw factory
Sep 15 11:25:05.254547 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254536    2574 manager.go:1196] Started watching for new ooms in manager
Sep 15 11:25:05.254838 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.254827    2574 manager.go:349] Starting recovery of all containers
Sep 15 11:25:05.261190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.260610    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 11:25:05.265172 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.265151    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:05.265564 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.265543    2574 manager.go:354] Recovery completed
Sep 15 11:25:05.267858 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.267833    2574 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-54.ec2.internal\" not found" node="ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.271656 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.271643    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.275939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.275922    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.276026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.275952    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.276026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.275963    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.276600 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.276584    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.276686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.276584    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 11:25:05.276686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.276613    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 11:25:05.276686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.276628    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 11:25:05.277335 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.277315    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.277408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.277348    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.277408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.277361    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.277760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.277745    2574 policy_none.go:49] "None policy: Start"
Sep 15 11:25:05.277829 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.277763    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 11:25:05.277829 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.277773    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 11:25:05.278838 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.278827    2574 policy_none.go:47] "Start"
Sep 15 11:25:05.314222 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314204    2574 manager.go:337] "Starting Device Plugin manager"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.314242    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314253    2574 server.go:87] "Starting device plugin registration server"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314591    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314602    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314701    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314769    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.314775    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.315297    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 11:25:05.322681 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.315341    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.360166 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.360142    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 11:25:05.360282 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.360191    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 11:25:05.360282 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.360217    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 11:25:05.360282 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.360255    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 11:25:05.362725 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.362707    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:05.414749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.414722    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.415729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.415709    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.415729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.415738    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.415898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.415751    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.415898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.415782    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.421612 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.421599    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.421671 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.421620    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-54.ec2.internal\": node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.431111 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.431065    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.460988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.460967    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"]
Sep 15 11:25:05.461050 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.461029    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.461941 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.461927    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.462001 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.461953    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.462001 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.461963    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.463211 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463199    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.463325 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463311    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.463363 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463338    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.463914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463894    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.464007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463903    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.464007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463950    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.464007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463962    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.464007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463924    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.464007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.463999    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.465107 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.465091    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.465184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.465121    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 11:25:05.465862 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.465846    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 11:25:05.465939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.465870    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 11:25:05.465939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.465880    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeHasSufficientPID"
Sep 15 11:25:05.493067 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.493048    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-54.ec2.internal\" not found" node="ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.497559 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.497544    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-54.ec2.internal\" not found" node="ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.532037 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.532004    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.556224 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.556197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/839baa2c846d8b905bce654725874020-config\") pod \"kube-apiserver-proxy-ip-10-0-142-54.ec2.internal\" (UID: \"839baa2c846d8b905bce654725874020\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.556224 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.556227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7735f558a747b75c959deecb90746d76-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal\" (UID: \"7735f558a747b75c959deecb90746d76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.556399 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.556250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7735f558a747b75c959deecb90746d76-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal\" (UID: \"7735f558a747b75c959deecb90746d76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.632704 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.632675    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.657126 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.657097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/839baa2c846d8b905bce654725874020-config\") pod \"kube-apiserver-proxy-ip-10-0-142-54.ec2.internal\" (UID: \"839baa2c846d8b905bce654725874020\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.657178 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.657109    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/839baa2c846d8b905bce654725874020-config\") pod \"kube-apiserver-proxy-ip-10-0-142-54.ec2.internal\" (UID: \"839baa2c846d8b905bce654725874020\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.657178 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.657160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7735f558a747b75c959deecb90746d76-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal\" (UID: \"7735f558a747b75c959deecb90746d76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.657238 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.657181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7735f558a747b75c959deecb90746d76-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal\" (UID: \"7735f558a747b75c959deecb90746d76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.657238 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.657206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7735f558a747b75c959deecb90746d76-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal\" (UID: \"7735f558a747b75c959deecb90746d76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.657300 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.657242    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7735f558a747b75c959deecb90746d76-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal\" (UID: \"7735f558a747b75c959deecb90746d76\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.733423 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.733363    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.796030 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.795996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.800930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:05.800912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:05.833743 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.833582    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:05.934110 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:05.934078    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.332674 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:07.332636    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.433230 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:07.433194    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.496694 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.496672    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 11:25:07.497100 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.496836    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 11:25:07.497100 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.496862    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 11:25:07.534067 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:07.534035    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.555150 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.555069    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 11:20:06 +0000 UTC" deadline="2028-03-13 06:26:44.189281793 +0000 UTC"
Sep 15 11:25:07.555150 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.555098    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13075h1m36.634186516s"
Sep 15 11:25:07.622047 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:07.622011    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7735f558a747b75c959deecb90746d76.slice/crio-dfe47232024c91be5afd5cdf7fae2fafdf8c45abc46022f553776249770bca50 WatchSource:0}: Error finding container dfe47232024c91be5afd5cdf7fae2fafdf8c45abc46022f553776249770bca50: Status 404 returned error can't find the container with id dfe47232024c91be5afd5cdf7fae2fafdf8c45abc46022f553776249770bca50
Sep 15 11:25:07.622320 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:07.622297    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839baa2c846d8b905bce654725874020.slice/crio-d4843f616741c0c46d964d9b1fd18dadf669a99ca07f430780f2d23653233be2 WatchSource:0}: Error finding container d4843f616741c0c46d964d9b1fd18dadf669a99ca07f430780f2d23653233be2: Status 404 returned error can't find the container with id d4843f616741c0c46d964d9b1fd18dadf669a99ca07f430780f2d23653233be2
Sep 15 11:25:07.625997 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.625983    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:25:07.634197 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:07.634175    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.661509 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.661450    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal" event={"ID":"7735f558a747b75c959deecb90746d76","Type":"ContainerStarted","Data":"dfe47232024c91be5afd5cdf7fae2fafdf8c45abc46022f553776249770bca50"}
Sep 15 11:25:07.662525 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.662492    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal" event={"ID":"839baa2c846d8b905bce654725874020","Type":"ContainerStarted","Data":"d4843f616741c0c46d964d9b1fd18dadf669a99ca07f430780f2d23653233be2"}
Sep 15 11:25:07.734709 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:07.734672    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.835295 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:07.835214    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-54.ec2.internal\" not found"
Sep 15 11:25:07.902371 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.902339    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:07.952216 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.952188    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:07.960730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.960707    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 11:25:07.962279 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.962264    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal"
Sep 15 11:25:07.971906 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:07.971884    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 11:25:08.140063 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.139977    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:08.375263 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.375237    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:08.458286 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.458217    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 11:25:08.533561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.533517    2574 apiserver.go:52] "Watching apiserver"
Sep 15 11:25:08.538551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.538530    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 11:25:08.540225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.540202    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-fdwrx","kube-system/konnectivity-agent-nwhzn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6","openshift-cluster-node-tuning-operator/tuned-tlddr","openshift-dns/node-resolver-jjhhp","openshift-image-registry/node-ca-gwcfc","openshift-network-operator/iptables-alerter-lxk2c","openshift-ovn-kubernetes/ovnkube-node-89gtx","kube-system/global-pull-secret-syncer-75nth","kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal","openshift-multus/multus-additional-cni-plugins-mdv4q","openshift-multus/multus-v6xrw","openshift-multus/network-metrics-daemon-4q24x"]
Sep 15 11:25:08.542585 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.542563    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.544721 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.544672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.544845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.544750    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.544845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.544768    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 11:25:08.544946 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.544935    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.545133 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.545113    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pm6k4\""
Sep 15 11:25:08.546865 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.546799    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 11:25:08.546959 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.546909    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g5ssh\""
Sep 15 11:25:08.547028 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.547002    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.547276 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.547100    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 11:25:08.547276 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.547174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.547402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.547176    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.549023 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.549005    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tns7t\""
Sep 15 11:25:08.549119 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.549076    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.549119 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.549076    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 11:25:08.549244 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.549097    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.549244 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.549216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.551006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.550965    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.551006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.550989    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 11:25:08.551154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.550966    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-x4ftj\""
Sep 15 11:25:08.551154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.551135    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.551662 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.551646    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.553457 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.553442    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qgddh\""
Sep 15 11:25:08.553457 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.553451    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.553633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.553618    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.554242 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.553993    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.556031 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.556010    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 11:25:08.556119 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.556049    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.556119 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.556107    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xtmnz\""
Sep 15 11:25:08.556228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.556155    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.556354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.556335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.556456 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.556438    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:08.558633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.558609    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:08.558715 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.558681    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:08.560710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.560684    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.563088 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94b561a0-f7ce-4dac-8f3e-19c071a0607e-host-slash\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.563188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563116    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-cni-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.563188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-cni-multus\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.563188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563156    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f4b9x\""
Sep 15 11:25:08.563188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-hostroot\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.563380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.563380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysconfig\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.563380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-sys\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.563380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwbfw\" (UniqueName: \"kubernetes.io/projected/6c9f072c-14e7-4808-b8a2-6ee04358d01d-kube-api-access-rwbfw\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.563380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563365    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 11:25:08.563640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563359    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87gp\" (UniqueName: \"kubernetes.io/projected/94b561a0-f7ce-4dac-8f3e-19c071a0607e-kube-api-access-g87gp\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.563640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-netns\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.563640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563588    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 11:25:08.563640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563578    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q92t5\" (UniqueName: \"kubernetes.io/projected/61022462-f64e-4e0b-a03e-22a751cae212-kube-api-access-q92t5\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.563640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563632    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnjq2\" (UniqueName: \"kubernetes.io/projected/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-kube-api-access-fnjq2\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.563879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysctl-conf\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.563879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563764    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-systemd\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.563879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563816    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61022462-f64e-4e0b-a03e-22a751cae212-tmp\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.563879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563848    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 11:25:08.563879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-cni-bin\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.564111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563908    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwr48\" (UniqueName: \"kubernetes.io/projected/0eb33362-b615-461f-b096-a6acdee64724-kube-api-access-lwr48\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.564111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-host\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.564111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.563995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-cnibin\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.564111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eb33362-b615-461f-b096-a6acdee64724-multus-daemon-config\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.564111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564070    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-device-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.564327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.564327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-sys-fs\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.564327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysctl-d\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.564732 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-k8s-cni-cncf-io\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.564831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-conf-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.564831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.564773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-multus-certs\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.565050 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-system-cni-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.565143 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565066    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-socket-dir-parent\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.565198 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565177    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-kubelet\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.565254 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59f7g\" (UniqueName: \"kubernetes.io/projected/285acd19-2f05-4e65-a29e-f44e7c1f84e6-kube-api-access-59f7g\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.565254 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565235    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-registration-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.565351 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-modprobe-d\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.565399 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:08.565450 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-lib-modules\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.565450 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-var-lib-kubelet\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.565588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eb33362-b615-461f-b096-a6acdee64724-cni-binary-copy\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.565588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-etc-kubernetes\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.565588 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.565563    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:08.565731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-kubernetes\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.565731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565631    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61022462-f64e-4e0b-a03e-22a751cae212-etc-tuned\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.565731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565676    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/285acd19-2f05-4e65-a29e-f44e7c1f84e6-hosts-file\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.565731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285acd19-2f05-4e65-a29e-f44e7c1f84e6-tmp-dir\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.565916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94b561a0-f7ce-4dac-8f3e-19c071a0607e-iptables-alerter-script\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.565916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565819    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-socket-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.565916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.565868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-run\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.566296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.566273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c9f072c-14e7-4808-b8a2-6ee04358d01d-host\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.566371 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.566321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c9f072c-14e7-4808-b8a2-6ee04358d01d-serviceca\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.568635 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.568617    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.570991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.570868    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 11:25:08.570991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.570881    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2wxf9\""
Sep 15 11:25:08.570991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.570897    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 11:25:08.571147 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.571052    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.571147 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.571138    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 11:25:08.571239 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.571178    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 11:25:08.571290 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.571261    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 11:25:08.571347 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.571331    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 11:25:08.573066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.573046    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 11:25:08.573311 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.573293    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 11:25:08.573422 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.573404    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vzgfm\""
Sep 15 11:25:08.581985 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.581963    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v5dl6"
Sep 15 11:25:08.587249 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.587229    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v5dl6"
Sep 15 11:25:08.653595 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.653571    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 11:25:08.666783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.666783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-system-cni-dir\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-system-cni-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-kubelet\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-etc-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-kubelet-config\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-registration-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666968    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-system-cni-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eb33362-b615-461f-b096-a6acdee64724-cni-binary-copy\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-etc-kubernetes\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.666997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-registration-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-kubelet\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-kubelet\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667073    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-var-lib-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667227    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-etc-kubernetes\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94b561a0-f7ce-4dac-8f3e-19c071a0607e-iptables-alerter-script\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4fsm\" (UniqueName: \"kubernetes.io/projected/d15d98ac-db1a-40e7-892a-ae31681023e1-kube-api-access-m4fsm\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667367    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.667472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-dbus\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-socket-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-run\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94b561a0-f7ce-4dac-8f3e-19c071a0607e-host-slash\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667660    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-hostroot\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-socket-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-log-socket\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/94b561a0-f7ce-4dac-8f3e-19c071a0607e-host-slash\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-ovnkube-config\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d15d98ac-db1a-40e7-892a-ae31681023e1-ovn-node-metrics-cert\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667807    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-hostroot\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667819    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-run\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667867    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0eb33362-b615-461f-b096-a6acdee64724-cni-binary-copy\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysconfig\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.667912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667905    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-sys\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysconfig\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwbfw\" (UniqueName: \"kubernetes.io/projected/6c9f072c-14e7-4808-b8a2-6ee04358d01d-kube-api-access-rwbfw\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667980    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-sys\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.667973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/94b561a0-f7ce-4dac-8f3e-19c071a0607e-iptables-alerter-script\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668006    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87gp\" (UniqueName: \"kubernetes.io/projected/94b561a0-f7ce-4dac-8f3e-19c071a0607e-kube-api-access-g87gp\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-netns\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwr48\" (UniqueName: \"kubernetes.io/projected/0eb33362-b615-461f-b096-a6acdee64724-kube-api-access-lwr48\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-netns\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnjw\" (UniqueName: \"kubernetes.io/projected/125ade64-8f23-480a-906c-4400b6463813-kube-api-access-nbnjw\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668120    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnjq2\" (UniqueName: \"kubernetes.io/projected/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-kube-api-access-fnjq2\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysctl-conf\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.668479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668284    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysctl-conf\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61022462-f64e-4e0b-a03e-22a751cae212-tmp\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-systemd-units\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668734    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-cni-netd\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54875eb0-167a-4e37-941a-7cde822004a2-konnectivity-ca\") pod \"konnectivity-agent-nwhzn\" (UID: \"54875eb0-167a-4e37-941a-7cde822004a2\") " pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-device-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.668983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysctl-d\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.669077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.669036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-device-dir\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.669480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.669414    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-etc-selinux\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.669596 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.669573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-k8s-cni-cncf-io\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.669680 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.669604    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 11:25:08.669680 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.669614    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-sysctl-d\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.669778 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.669677    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-k8s-cni-cncf-io\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-conf-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-multus-certs\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr4qd\" (UniqueName: \"kubernetes.io/projected/acfbfdf0-00c1-4101-a771-797fe7db1917-kube-api-access-hr4qd\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-socket-dir-parent\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-cnibin\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59f7g\" (UniqueName: \"kubernetes.io/projected/285acd19-2f05-4e65-a29e-f44e7c1f84e6-kube-api-access-59f7g\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-modprobe-d\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-lib-modules\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-var-lib-kubelet\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-cni-bin\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-kubernetes\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61022462-f64e-4e0b-a03e-22a751cae212-etc-tuned\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670654    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/285acd19-2f05-4e65-a29e-f44e7c1f84e6-hosts-file\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670663    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-conf-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285acd19-2f05-4e65-a29e-f44e7c1f84e6-tmp-dir\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670744    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-run-multus-certs\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670790    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-lib-modules\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c9f072c-14e7-4808-b8a2-6ee04358d01d-host\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c9f072c-14e7-4808-b8a2-6ee04358d01d-serviceca\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670869    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-socket-dir-parent\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-cni-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-cni-multus\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670945    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-modprobe-d\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54875eb0-167a-4e37-941a-7cde822004a2-agent-certs\") pod \"konnectivity-agent-nwhzn\" (UID: \"54875eb0-167a-4e37-941a-7cde822004a2\") " pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/285acd19-2f05-4e65-a29e-f44e7c1f84e6-tmp-dir\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.670979    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-run-netns\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-kubernetes\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671038    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6c9f072c-14e7-4808-b8a2-6ee04358d01d-host\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-node-log\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671118    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-var-lib-kubelet\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671238    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-multus-cni-dir\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671316    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-cni-multus\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/285acd19-2f05-4e65-a29e-f44e7c1f84e6-hosts-file\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.673584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-env-overrides\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q92t5\" (UniqueName: \"kubernetes.io/projected/61022462-f64e-4e0b-a03e-22a751cae212-kube-api-access-q92t5\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671604    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-systemd\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6c9f072c-14e7-4808-b8a2-6ee04358d01d-serviceca\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-cni-bin\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671704    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-etc-systemd\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-host\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-cnibin\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eb33362-b615-461f-b096-a6acdee64724-multus-daemon-config\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-ovn\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.671976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-ovnkube-script-lib\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61022462-f64e-4e0b-a03e-22a751cae212-host\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-sys-fs\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.674416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-slash\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.675134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-systemd\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.675134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672365    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-host-var-lib-cni-bin\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.675134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672437    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-sys-fs\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.675134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672509    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0eb33362-b615-461f-b096-a6acdee64724-cnibin\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.675134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.672558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0eb33362-b615-461f-b096-a6acdee64724-multus-daemon-config\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.675371 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.675260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61022462-f64e-4e0b-a03e-22a751cae212-etc-tuned\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.675622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.675579    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61022462-f64e-4e0b-a03e-22a751cae212-tmp\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.676781 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.676528    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87gp\" (UniqueName: \"kubernetes.io/projected/94b561a0-f7ce-4dac-8f3e-19c071a0607e-kube-api-access-g87gp\") pod \"iptables-alerter-lxk2c\" (UID: \"94b561a0-f7ce-4dac-8f3e-19c071a0607e\") " pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.677044 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.677023    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwbfw\" (UniqueName: \"kubernetes.io/projected/6c9f072c-14e7-4808-b8a2-6ee04358d01d-kube-api-access-rwbfw\") pod \"node-ca-gwcfc\" (UID: \"6c9f072c-14e7-4808-b8a2-6ee04358d01d\") " pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.678434 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.678398    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59f7g\" (UniqueName: \"kubernetes.io/projected/285acd19-2f05-4e65-a29e-f44e7c1f84e6-kube-api-access-59f7g\") pod \"node-resolver-jjhhp\" (UID: \"285acd19-2f05-4e65-a29e-f44e7c1f84e6\") " pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.678434 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.678415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnjq2\" (UniqueName: \"kubernetes.io/projected/881ae06b-b2d0-43cf-a6ac-f9a56a1db819-kube-api-access-fnjq2\") pod \"aws-ebs-csi-driver-node-kcnd6\" (UID: \"881ae06b-b2d0-43cf-a6ac-f9a56a1db819\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.678789 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.678772    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q92t5\" (UniqueName: \"kubernetes.io/projected/61022462-f64e-4e0b-a03e-22a751cae212-kube-api-access-q92t5\") pod \"tuned-tlddr\" (UID: \"61022462-f64e-4e0b-a03e-22a751cae212\") " pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.679350 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.679329    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwr48\" (UniqueName: \"kubernetes.io/projected/0eb33362-b615-461f-b096-a6acdee64724-kube-api-access-lwr48\") pod \"multus-v6xrw\" (UID: \"0eb33362-b615-461f-b096-a6acdee64724\") " pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.773564 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-slash\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773564 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-systemd\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773564 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773565    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-system-cni-dir\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773591    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-etc-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773600    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-systemd\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-slash\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-system-cni-dir\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773658    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-kubelet-config\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-etc-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-kubelet\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773736    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-kubelet-config\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773764    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-var-lib-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4fsm\" (UniqueName: \"kubernetes.io/projected/d15d98ac-db1a-40e7-892a-ae31681023e1-kube-api-access-m4fsm\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773800    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-kubelet\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.773837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773849    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-var-lib-openvswitch\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773857    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-dbus\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773907    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-log-socket\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-ovnkube-config\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d15d98ac-db1a-40e7-892a-ae31681023e1-ovn-node-metrics-cert\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.773996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnjw\" (UniqueName: \"kubernetes.io/projected/125ade64-8f23-480a-906c-4400b6463813-kube-api-access-nbnjw\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774015    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-systemd-units\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-cni-netd\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-cni-netd\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774162    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-systemd-units\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-dbus\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774283    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-log-socket\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54875eb0-167a-4e37-941a-7cde822004a2-konnectivity-ca\") pod \"konnectivity-agent-nwhzn\" (UID: \"54875eb0-167a-4e37-941a-7cde822004a2\") " pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr4qd\" (UniqueName: \"kubernetes.io/projected/acfbfdf0-00c1-4101-a771-797fe7db1917-kube-api-access-hr4qd\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.774668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774395    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-cnibin\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-cni-bin\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774454    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774526    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/acfbfdf0-00c1-4101-a771-797fe7db1917-cnibin\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774571    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-cni-bin\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774651    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-ovnkube-config\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.774662    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54875eb0-167a-4e37-941a-7cde822004a2-agent-certs\") pod \"konnectivity-agent-nwhzn\" (UID: \"54875eb0-167a-4e37-941a-7cde822004a2\") " pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.774740    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:09.274703398 +0000 UTC m=+3.052379652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-run-netns\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-node-log\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-env-overrides\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774912    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-node-log\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:08.775515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774969    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.774996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-ovn\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.775009    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-ovnkube-script-lib\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.775054    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:09.275039273 +0000 UTC m=+3.052715522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775135    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/54875eb0-167a-4e37-941a-7cde822004a2-konnectivity-ca\") pod \"konnectivity-agent-nwhzn\" (UID: \"54875eb0-167a-4e37-941a-7cde822004a2\") " pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-run-ovn-kubernetes\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-run-ovn\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d15d98ac-db1a-40e7-892a-ae31681023e1-host-run-netns\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-env-overrides\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775447    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775581    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d15d98ac-db1a-40e7-892a-ae31681023e1-ovnkube-script-lib\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.776345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.775587    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/acfbfdf0-00c1-4101-a771-797fe7db1917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.776933 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.776878    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d15d98ac-db1a-40e7-892a-ae31681023e1-ovn-node-metrics-cert\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.777830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.777798    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/54875eb0-167a-4e37-941a-7cde822004a2-agent-certs\") pod \"konnectivity-agent-nwhzn\" (UID: \"54875eb0-167a-4e37-941a-7cde822004a2\") " pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.784311 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.784289    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:08.784419 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.784318    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:08.784419 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.784330    2574 projected.go:196] Error preparing data for projected volume kube-api-access-82bw8 for pod openshift-network-diagnostics/network-check-target-fdwrx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:08.784419 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:08.784402    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8 podName:86729383-b3ac-4559-b996-929287275e55 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:09.284385202 +0000 UTC m=+3.062061464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-82bw8" (UniqueName: "kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8") pod "network-check-target-fdwrx" (UID: "86729383-b3ac-4559-b996-929287275e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:08.786519 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.786464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnjw\" (UniqueName: \"kubernetes.io/projected/125ade64-8f23-480a-906c-4400b6463813-kube-api-access-nbnjw\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:08.786953 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.786929    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4fsm\" (UniqueName: \"kubernetes.io/projected/d15d98ac-db1a-40e7-892a-ae31681023e1-kube-api-access-m4fsm\") pod \"ovnkube-node-89gtx\" (UID: \"d15d98ac-db1a-40e7-892a-ae31681023e1\") " pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.787026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.786984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr4qd\" (UniqueName: \"kubernetes.io/projected/acfbfdf0-00c1-4101-a771-797fe7db1917-kube-api-access-hr4qd\") pod \"multus-additional-cni-plugins-mdv4q\" (UID: \"acfbfdf0-00c1-4101-a771-797fe7db1917\") " pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:08.855371 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.855334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lxk2c"
Sep 15 11:25:08.862622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.862593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v6xrw"
Sep 15 11:25:08.872733 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.872712    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6"
Sep 15 11:25:08.879171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.879153    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tlddr"
Sep 15 11:25:08.884847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.884832    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jjhhp"
Sep 15 11:25:08.890923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.890907    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gwcfc"
Sep 15 11:25:08.897077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.897056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:08.902755 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.902739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:08.908393 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:08.908373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdv4q"
Sep 15 11:25:09.279045 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.279013    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:09.279217 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.279082    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:09.279217 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.279190    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:09.279327 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.279225    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:09.279327 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.279253    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:10.279233932 +0000 UTC m=+4.056910174 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:09.279327 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.279288    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:10.279269895 +0000 UTC m=+4.056946150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:09.379692 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.379658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:09.379867 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.379814    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:09.379867 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.379835    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:09.379867 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.379847    2574 projected.go:196] Error preparing data for projected volume kube-api-access-82bw8 for pod openshift-network-diagnostics/network-check-target-fdwrx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:09.380000 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.379905    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8 podName:86729383-b3ac-4559-b996-929287275e55 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:10.379886687 +0000 UTC m=+4.157562922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-82bw8" (UniqueName: "kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8") pod "network-check-target-fdwrx" (UID: "86729383-b3ac-4559-b996-929287275e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:09.490387 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.490222    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b561a0_f7ce_4dac_8f3e_19c071a0607e.slice/crio-3493f301291d3383323ab4dd9793bd20246fbf419444cb6cbc5a2ef7126eb964 WatchSource:0}: Error finding container 3493f301291d3383323ab4dd9793bd20246fbf419444cb6cbc5a2ef7126eb964: Status 404 returned error can't find the container with id 3493f301291d3383323ab4dd9793bd20246fbf419444cb6cbc5a2ef7126eb964
Sep 15 11:25:09.491830 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.491732    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb33362_b615_461f_b096_a6acdee64724.slice/crio-cc27c23a8da6f4c8de3d54c9b3b907bc7eda8ca6d9b633fa72c5d094e4742513 WatchSource:0}: Error finding container cc27c23a8da6f4c8de3d54c9b3b907bc7eda8ca6d9b633fa72c5d094e4742513: Status 404 returned error can't find the container with id cc27c23a8da6f4c8de3d54c9b3b907bc7eda8ca6d9b633fa72c5d094e4742513
Sep 15 11:25:09.494021 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.493983    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285acd19_2f05_4e65_a29e_f44e7c1f84e6.slice/crio-083a0d563c4c85222338658155e704b497cc7bfbc9259c6b31a047a810103969 WatchSource:0}: Error finding container 083a0d563c4c85222338658155e704b497cc7bfbc9259c6b31a047a810103969: Status 404 returned error can't find the container with id 083a0d563c4c85222338658155e704b497cc7bfbc9259c6b31a047a810103969
Sep 15 11:25:09.494632 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.494609    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61022462_f64e_4e0b_a03e_22a751cae212.slice/crio-ecbcd8fc3d831a9d89a662eb1a8b8382b3422d84e101c6591a59d929e33a0704 WatchSource:0}: Error finding container ecbcd8fc3d831a9d89a662eb1a8b8382b3422d84e101c6591a59d929e33a0704: Status 404 returned error can't find the container with id ecbcd8fc3d831a9d89a662eb1a8b8382b3422d84e101c6591a59d929e33a0704
Sep 15 11:25:09.496384 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.496362    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881ae06b_b2d0_43cf_a6ac_f9a56a1db819.slice/crio-7c5b9d443427b0c2ffbb99cbb72b634247e4cf1a930c3fb2775d5ba3eeef7f07 WatchSource:0}: Error finding container 7c5b9d443427b0c2ffbb99cbb72b634247e4cf1a930c3fb2775d5ba3eeef7f07: Status 404 returned error can't find the container with id 7c5b9d443427b0c2ffbb99cbb72b634247e4cf1a930c3fb2775d5ba3eeef7f07
Sep 15 11:25:09.497259 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.497232    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfbfdf0_00c1_4101_a771_797fe7db1917.slice/crio-5e56839ea7b9984fffea5c1347e663d95e8b256633601c36e1379b65f1ede398 WatchSource:0}: Error finding container 5e56839ea7b9984fffea5c1347e663d95e8b256633601c36e1379b65f1ede398: Status 404 returned error can't find the container with id 5e56839ea7b9984fffea5c1347e663d95e8b256633601c36e1379b65f1ede398
Sep 15 11:25:09.498049 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.498022    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9f072c_14e7_4808_b8a2_6ee04358d01d.slice/crio-90e3b458918603454de336b74e9874cdb0827af0ac7d254b7e8a2d65060955fa WatchSource:0}: Error finding container 90e3b458918603454de336b74e9874cdb0827af0ac7d254b7e8a2d65060955fa: Status 404 returned error can't find the container with id 90e3b458918603454de336b74e9874cdb0827af0ac7d254b7e8a2d65060955fa
Sep 15 11:25:09.499558 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.499107    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15d98ac_db1a_40e7_892a_ae31681023e1.slice/crio-571d5dda26de9979b821a21390b7c05517783941273a32882201257a3472cac6 WatchSource:0}: Error finding container 571d5dda26de9979b821a21390b7c05517783941273a32882201257a3472cac6: Status 404 returned error can't find the container with id 571d5dda26de9979b821a21390b7c05517783941273a32882201257a3472cac6
Sep 15 11:25:09.500780 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:09.500640    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54875eb0_167a_4e37_941a_7cde822004a2.slice/crio-3b4dabb36339e5f37bb1b81641a9e85f9296648f3775ba78c13e5bfaf797b496 WatchSource:0}: Error finding container 3b4dabb36339e5f37bb1b81641a9e85f9296648f3775ba78c13e5bfaf797b496: Status 404 returned error can't find the container with id 3b4dabb36339e5f37bb1b81641a9e85f9296648f3775ba78c13e5bfaf797b496
Sep 15 11:25:09.590314 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.590270    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:20:08 +0000 UTC" deadline="2028-05-26 19:17:11.908555616 +0000 UTC"
Sep 15 11:25:09.590314 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.590299    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14863h52m2.31826017s"
Sep 15 11:25:09.658747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.658721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:09.658902 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:09.658859    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:09.669668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.669634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal" event={"ID":"839baa2c846d8b905bce654725874020","Type":"ContainerStarted","Data":"356935ee70b50a20eb07c3014552a264983ac43272be1bf8000991519cf6a6bd"}
Sep 15 11:25:09.670731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.670708    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nwhzn" event={"ID":"54875eb0-167a-4e37-941a-7cde822004a2","Type":"ContainerStarted","Data":"3b4dabb36339e5f37bb1b81641a9e85f9296648f3775ba78c13e5bfaf797b496"}
Sep 15 11:25:09.673183 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.673159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"571d5dda26de9979b821a21390b7c05517783941273a32882201257a3472cac6"}
Sep 15 11:25:09.674959 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.674935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gwcfc" event={"ID":"6c9f072c-14e7-4808-b8a2-6ee04358d01d","Type":"ContainerStarted","Data":"90e3b458918603454de336b74e9874cdb0827af0ac7d254b7e8a2d65060955fa"}
Sep 15 11:25:09.676193 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.676168    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerStarted","Data":"5e56839ea7b9984fffea5c1347e663d95e8b256633601c36e1379b65f1ede398"}
Sep 15 11:25:09.677354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.677337    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v6xrw" event={"ID":"0eb33362-b615-461f-b096-a6acdee64724","Type":"ContainerStarted","Data":"cc27c23a8da6f4c8de3d54c9b3b907bc7eda8ca6d9b633fa72c5d094e4742513"}
Sep 15 11:25:09.678731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.678711    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lxk2c" event={"ID":"94b561a0-f7ce-4dac-8f3e-19c071a0607e","Type":"ContainerStarted","Data":"3493f301291d3383323ab4dd9793bd20246fbf419444cb6cbc5a2ef7126eb964"}
Sep 15 11:25:09.680090 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.680072    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6" event={"ID":"881ae06b-b2d0-43cf-a6ac-f9a56a1db819","Type":"ContainerStarted","Data":"7c5b9d443427b0c2ffbb99cbb72b634247e4cf1a930c3fb2775d5ba3eeef7f07"}
Sep 15 11:25:09.681058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.681028    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tlddr" event={"ID":"61022462-f64e-4e0b-a03e-22a751cae212","Type":"ContainerStarted","Data":"ecbcd8fc3d831a9d89a662eb1a8b8382b3422d84e101c6591a59d929e33a0704"}
Sep 15 11:25:09.682342 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.682294    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-54.ec2.internal" podStartSLOduration=2.682278949 podStartE2EDuration="2.682278949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:09.681886339 +0000 UTC m=+3.459562596" watchObservedRunningTime="2026-09-15 11:25:09.682278949 +0000 UTC m=+3.459955205"
Sep 15 11:25:09.683273 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:09.683254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jjhhp" event={"ID":"285acd19-2f05-4e65-a29e-f44e7c1f84e6","Type":"ContainerStarted","Data":"083a0d563c4c85222338658155e704b497cc7bfbc9259c6b31a047a810103969"}
Sep 15 11:25:10.287479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.287173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:10.287800 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.287516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:10.287800 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.287672    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:10.287800 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.287736    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:12.287716073 +0000 UTC m=+6.065392328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:10.288171 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.288151    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:10.288253 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.288212    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:12.288195848 +0000 UTC m=+6.065872097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:10.389084 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.388412    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:10.389084 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.388629    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:10.389084 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.388650    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:10.389084 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.388662    2574 projected.go:196] Error preparing data for projected volume kube-api-access-82bw8 for pod openshift-network-diagnostics/network-check-target-fdwrx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:10.389084 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.388717    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8 podName:86729383-b3ac-4559-b996-929287275e55 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:12.388697391 +0000 UTC m=+6.166373640 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-82bw8" (UniqueName: "kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8") pod "network-check-target-fdwrx" (UID: "86729383-b3ac-4559-b996-929287275e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:10.590623 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.590461    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 11:20:08 +0000 UTC" deadline="2028-05-10 23:46:15.704966733 +0000 UTC"
Sep 15 11:25:10.590623 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.590542    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14484h21m5.114429588s"
Sep 15 11:25:10.660957 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.660918    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:10.661116 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.661062    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:10.661484 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.661466    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:10.661596 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:10.661577    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:10.694276 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.694238    2574 generic.go:356] "Generic (PLEG): container finished" podID="7735f558a747b75c959deecb90746d76" containerID="0503fb7838e0cd990002d9e3a98ea7fc3ff81d388e143187068b394a673208cf" exitCode=0
Sep 15 11:25:10.695240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:10.695212    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal" event={"ID":"7735f558a747b75c959deecb90746d76","Type":"ContainerDied","Data":"0503fb7838e0cd990002d9e3a98ea7fc3ff81d388e143187068b394a673208cf"}
Sep 15 11:25:11.658761 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:11.658729    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:11.659199 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:11.658860    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:11.706748 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:11.706712    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal" event={"ID":"7735f558a747b75c959deecb90746d76","Type":"ContainerStarted","Data":"e7db48ae69b8a5017af87d620d790dfdd6fbad9f70aa834f60866670633f0c31"}
Sep 15 11:25:12.307482 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:12.306869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:12.307482 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:12.306949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:12.307482 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.307093    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:12.307482 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.307118    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:12.307482 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.307154    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:16.3071374 +0000 UTC m=+10.084813642 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:12.307482 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.307183    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:16.307165315 +0000 UTC m=+10.084841571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:12.407348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:12.407312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:12.407556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.407468    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:12.407556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.407488    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:12.407556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.407514    2574 projected.go:196] Error preparing data for projected volume kube-api-access-82bw8 for pod openshift-network-diagnostics/network-check-target-fdwrx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:12.407723 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.407589    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8 podName:86729383-b3ac-4559-b996-929287275e55 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:16.407569536 +0000 UTC m=+10.185245790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-82bw8" (UniqueName: "kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8") pod "network-check-target-fdwrx" (UID: "86729383-b3ac-4559-b996-929287275e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:12.663556 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:12.659561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:12.663556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.659755    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:12.666890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:12.664603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:12.666890 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:12.664815    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:13.659505 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:13.659462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:13.659693 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:13.659622    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:14.662206 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:14.662176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:14.662206 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:14.662205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:14.662708 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:14.662300    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:14.662708 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:14.662447    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:15.659267 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:15.659102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:15.659267 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:15.659237    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:16.343106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:16.342577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:16.343106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:16.342657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:16.343106 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.342741    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:16.343106 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.342781    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:16.343106 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.342816    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:24.34279659 +0000 UTC m=+18.120472840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:16.343106 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.342835    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:24.34282557 +0000 UTC m=+18.120501811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:16.443644 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:16.443571    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:16.443811 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.443764    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:16.443811 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.443791    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:16.443811 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.443801    2574 projected.go:196] Error preparing data for projected volume kube-api-access-82bw8 for pod openshift-network-diagnostics/network-check-target-fdwrx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:16.443983 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.443869    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8 podName:86729383-b3ac-4559-b996-929287275e55 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:24.443844719 +0000 UTC m=+18.221520978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-82bw8" (UniqueName: "kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8") pod "network-check-target-fdwrx" (UID: "86729383-b3ac-4559-b996-929287275e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:16.660415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:16.659885    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:16.660415 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.659999    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:16.660415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:16.660047    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:16.660415 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:16.660160    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:17.659482 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:17.659442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:17.659968 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:17.659602    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:18.658954 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:18.658921    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:18.659126 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:18.658958    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:18.659126 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:18.659078    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:18.659257 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:18.659180    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:19.658777 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:19.658736    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:19.659218 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:19.658867    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:20.659469 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:20.659431    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:20.659844 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:20.659445    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:20.659844 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:20.659587    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:20.659844 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:20.659611    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:21.659236 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:21.659205    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:21.659419 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:21.659319    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:22.659412 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:22.659374    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:22.659924 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:22.659527    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:22.659924 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:22.659562    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:22.659924 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:22.659672    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:23.658682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:23.658639    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:23.658906 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:23.658773    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:24.404588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:24.404545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:24.405071 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:24.404613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:24.405071 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.404697    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:24.405071 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.404756    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:40.404740887 +0000 UTC m=+34.182417142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:24.405071 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.404697    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:24.405071 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.404861    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:40.40483838 +0000 UTC m=+34.182514617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:24.505688 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:24.505658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:24.505859 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.505840    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 11:25:24.505907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.505865    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 11:25:24.505907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.505875    2574 projected.go:196] Error preparing data for projected volume kube-api-access-82bw8 for pod openshift-network-diagnostics/network-check-target-fdwrx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:24.505994 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.505937    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8 podName:86729383-b3ac-4559-b996-929287275e55 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:40.505921154 +0000 UTC m=+34.283597394 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-82bw8" (UniqueName: "kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8") pod "network-check-target-fdwrx" (UID: "86729383-b3ac-4559-b996-929287275e55") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 11:25:24.659436 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:24.659344    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:24.659436 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:24.659367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:24.659664 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.659485    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:24.659664 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:24.659601    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:25.659171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:25.659139    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:25.659602 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:25.659255    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:26.659690 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:26.659601    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:26.660085 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:26.659713    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:26.660085 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:26.659770    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:26.660085 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:26.659855    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:27.659359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.658992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:27.659549 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:27.659522    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:27.743527 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.743482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jjhhp" event={"ID":"285acd19-2f05-4e65-a29e-f44e7c1f84e6","Type":"ContainerStarted","Data":"07979e734c6c54f256c6a43a04b1d0e806ed3712b3517715f5414c28b5c996ce"}
Sep 15 11:25:27.745114 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.745087    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nwhzn" event={"ID":"54875eb0-167a-4e37-941a-7cde822004a2","Type":"ContainerStarted","Data":"3a98338e9ab65eb2bb3c7bfb7b20a49223daeda2a239c40ca4a3d991d96d9bd7"}
Sep 15 11:25:27.747747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.747723    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:25:27.748120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.748099    2574 generic.go:356] "Generic (PLEG): container finished" podID="d15d98ac-db1a-40e7-892a-ae31681023e1" containerID="a1629efdaf0e531cf6febf62a6e6313344b15c2bf40f9ed3b943bce84e8c2480" exitCode=1
Sep 15 11:25:27.748199 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.748175    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"b4c5b1119ba8cf3e2ab092b9988f3c50a8e3c4e247e0fbc0b2f2cad514584473"}
Sep 15 11:25:27.748275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.748259    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"e6cddde610f06a30d3ae5e2867fe6be139f22c019bebc7a4ee669b82ae88798b"}
Sep 15 11:25:27.748329 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.748315    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerDied","Data":"a1629efdaf0e531cf6febf62a6e6313344b15c2bf40f9ed3b943bce84e8c2480"}
Sep 15 11:25:27.748386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.748373    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"08d33bbc9e84fffdab4b7a16f813f7a2c81049e40b14d92d422b87318df3bc2e"}
Sep 15 11:25:27.749815 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.749789    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gwcfc" event={"ID":"6c9f072c-14e7-4808-b8a2-6ee04358d01d","Type":"ContainerStarted","Data":"3b0bccee639b7da92ac1879b0deeaf33997a54f63656ee103a3cbc4be8e037cb"}
Sep 15 11:25:27.751793 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.751765    2574 generic.go:356] "Generic (PLEG): container finished" podID="acfbfdf0-00c1-4101-a771-797fe7db1917" containerID="10b8f24459e817036bb4dd83049181d8bd10bd4008d84a0b2ae709abf90622eb" exitCode=0
Sep 15 11:25:27.751873 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.751837    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerDied","Data":"10b8f24459e817036bb4dd83049181d8bd10bd4008d84a0b2ae709abf90622eb"}
Sep 15 11:25:27.753489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.753465    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v6xrw" event={"ID":"0eb33362-b615-461f-b096-a6acdee64724","Type":"ContainerStarted","Data":"5ad11212da3d60436a882b3266db062bd5fa6d5e59e91a0e52de4a7bef49f496"}
Sep 15 11:25:27.755039 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.755021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6" event={"ID":"881ae06b-b2d0-43cf-a6ac-f9a56a1db819","Type":"ContainerStarted","Data":"125ae37dcaf4009dca84ae0e6b23a803b7942ac5c3b887376b5c8229856b5712"}
Sep 15 11:25:27.756308 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.756293    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tlddr" event={"ID":"61022462-f64e-4e0b-a03e-22a751cae212","Type":"ContainerStarted","Data":"302a96eb80acc8c455db38261393a4ec00bb8cea52bc4eba22927702b86f5e26"}
Sep 15 11:25:27.757983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.757943    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-54.ec2.internal" podStartSLOduration=20.757929616 podStartE2EDuration="20.757929616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:11.7209565 +0000 UTC m=+5.498632760" watchObservedRunningTime="2026-09-15 11:25:27.757929616 +0000 UTC m=+21.535605874"
Sep 15 11:25:27.758383 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.758355    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jjhhp" podStartSLOduration=3.977999558 podStartE2EDuration="21.758348812s" totalImagesPullingTime="17.780349254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:27.757451172 +0000 UTC m=+21.535127439" watchObservedRunningTime="2026-09-15 11:25:27.758348812 +0000 UTC m=+21.536025069"
Sep 15 11:25:27.770052 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.770015    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nwhzn" podStartSLOduration=5.054914874 podStartE2EDuration="21.770004876s" totalImagesPullingTime="16.715090002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:27.769869517 +0000 UTC m=+21.547545774" watchObservedRunningTime="2026-09-15 11:25:27.770004876 +0000 UTC m=+21.547681135"
Sep 15 11:25:27.807038 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.806999    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gwcfc" podStartSLOduration=4.0390349 podStartE2EDuration="21.806983418s" totalImagesPullingTime="17.767948518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:27.788578342 +0000 UTC m=+21.566254600" watchObservedRunningTime="2026-09-15 11:25:27.806983418 +0000 UTC m=+21.584659674"
Sep 15 11:25:27.822337 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.822295    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v6xrw" podStartSLOduration=3.804588028 podStartE2EDuration="21.822280391s" totalImagesPullingTime="18.017692363s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:27.821986071 +0000 UTC m=+21.599662328" watchObservedRunningTime="2026-09-15 11:25:27.822280391 +0000 UTC m=+21.599956648"
Sep 15 11:25:27.840533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:27.838279    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tlddr" podStartSLOduration=4.057403846 podStartE2EDuration="21.838261062s" totalImagesPullingTime="17.780857216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:27.83755674 +0000 UTC m=+21.615232996" watchObservedRunningTime="2026-09-15 11:25:27.838261062 +0000 UTC m=+21.615937319"
Sep 15 11:25:28.641879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.641832    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 11:25:28.659374 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.659350    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:28.659530 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.659375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:28.659530 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:28.659477    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:28.659654 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:28.659595    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:28.762250 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.762228    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:25:28.762787 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.762758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"9a37d4d7205529873f518c054aa1a38501b18c66039eba5b24858c6357437044"}
Sep 15 11:25:28.762894 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.762843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"fde59d9e11b3e8750e359da4d221e6f689ae68bac255551a72a3a07e971f5e28"}
Sep 15 11:25:28.764400 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.764368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lxk2c" event={"ID":"94b561a0-f7ce-4dac-8f3e-19c071a0607e","Type":"ContainerStarted","Data":"05ef5ecb3e3439ae4b5be971538f81a21bd24c531acbee0585f68100464e8d5a"}
Sep 15 11:25:28.766776 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:28.766552    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6" event={"ID":"881ae06b-b2d0-43cf-a6ac-f9a56a1db819","Type":"ContainerStarted","Data":"3def9c5ee34b92811ac1857e8b0b8533724c2c9cb9c03829ca48c87bc55cb89c"}
Sep 15 11:25:29.623993 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:29.623881    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T11:25:28.641854815Z","UUID":"eb0070c6-6a7e-4d7c-9f3a-50b1d0a7bbef","Handler":null,"Name":"","Endpoint":""}
Sep 15 11:25:29.625584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:29.625562    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 11:25:29.625703 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:29.625602    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 11:25:29.658929 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:29.658896    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:29.659105 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:29.659050    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:29.771689 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:29.771643    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6" event={"ID":"881ae06b-b2d0-43cf-a6ac-f9a56a1db819","Type":"ContainerStarted","Data":"c7bfca76fb96dae6a96335c915714fb523f2c5cb60e444f8617ab76bb4e4ed75"}
Sep 15 11:25:29.789861 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:29.789814    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lxk2c" podStartSLOduration=6.014072612 podStartE2EDuration="23.789798121s" totalImagesPullingTime="17.775725509s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:28.778186391 +0000 UTC m=+22.555862650" watchObservedRunningTime="2026-09-15 11:25:29.789798121 +0000 UTC m=+23.567474386"
Sep 15 11:25:30.659343 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:30.659307    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:30.659343 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:30.659335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:30.659692 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:30.659445    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:30.659692 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:30.659558    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:30.777709 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:30.777682    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:25:30.778160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:30.778140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"1a9e8d74cc796784a2f0b80078f2b2daf6dd0f939a712f4ec25abede53c3443f"}
Sep 15 11:25:31.223557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:31.223355    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:31.354782 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:31.354741    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:31.355907 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:31.355879    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:31.369195 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:31.369149    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kcnd6" podStartSLOduration=5.638729754 podStartE2EDuration="25.369135087s" totalImagesPullingTime="19.730405333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:29.789586234 +0000 UTC m=+23.567262492" watchObservedRunningTime="2026-09-15 11:25:31.369135087 +0000 UTC m=+25.146811358"
Sep 15 11:25:31.658905 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:31.658830    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:31.659071 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:31.658945    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:31.780967 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:31.780941    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nwhzn"
Sep 15 11:25:32.659279 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.659202    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:32.659414 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:32.659327    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:32.659414 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.659371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:32.659493 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:32.659450    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:32.785440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.785412    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:25:32.786195 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.785803    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"c076a569a4fc996e588e6b8b4cfcf906ec1de3ab2ba23320b2508aee0d432609"}
Sep 15 11:25:32.786195 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.786076    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:32.786327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.786296    2574 scope.go:117] "RemoveContainer" containerID="a1629efdaf0e531cf6febf62a6e6313344b15c2bf40f9ed3b943bce84e8c2480"
Sep 15 11:25:32.787845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.787821    2574 generic.go:356] "Generic (PLEG): container finished" podID="acfbfdf0-00c1-4101-a771-797fe7db1917" containerID="0bf75b9eb086e9da244041e277be35382a1fecb9115564cfd27d82814199f256" exitCode=0
Sep 15 11:25:32.787975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.787900    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerDied","Data":"0bf75b9eb086e9da244041e277be35382a1fecb9115564cfd27d82814199f256"}
Sep 15 11:25:32.804997 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:32.804969    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:33.658752 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.658724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:33.658909 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:33.658826    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:33.793748 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.793723    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:25:33.794125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.794102    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" event={"ID":"d15d98ac-db1a-40e7-892a-ae31681023e1","Type":"ContainerStarted","Data":"f875e2c8533ef75499a86fda1ceba829ecfd6c5af3c1e2af4431c88a2189c380"}
Sep 15 11:25:33.794533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.794512    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:33.794646 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.794585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:33.809389 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.809365    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:25:33.820376 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:33.820282    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx" podStartSLOduration=9.950356181 podStartE2EDuration="27.820263563s" totalImagesPullingTime="17.869907382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:33.819628594 +0000 UTC m=+27.597304871" watchObservedRunningTime="2026-09-15 11:25:33.820263563 +0000 UTC m=+27.597939852"
Sep 15 11:25:34.082636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:34.082562    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-75nth"]
Sep 15 11:25:34.082785 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:34.082732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:34.082867 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:34.082842    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:34.085472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:34.085449    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4q24x"]
Sep 15 11:25:34.085659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:34.085562    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:34.085659 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:34.085641    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:34.086097 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:34.086076    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fdwrx"]
Sep 15 11:25:34.086191 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:34.086177    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:34.086311 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:34.086284    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:35.659535 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:35.659418    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:35.659535 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:35.659449    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:35.659535 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:35.659469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:35.659981 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:35.659573    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:35.659981 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:35.659638    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:35.659981 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:35.659712    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:35.800686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:35.800653    2574 generic.go:356] "Generic (PLEG): container finished" podID="acfbfdf0-00c1-4101-a771-797fe7db1917" containerID="2f81ab1d827c85a275219cd201521ff91eca41083ee7ef8905269da50942b85e" exitCode=0
Sep 15 11:25:35.800828 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:35.800741    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerDied","Data":"2f81ab1d827c85a275219cd201521ff91eca41083ee7ef8905269da50942b85e"}
Sep 15 11:25:37.659475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:37.659303    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:37.659845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:37.659303    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:37.659845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:37.659303    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:37.659845 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:37.659574    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:37.659845 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:37.659646    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:37.659845 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:37.659695    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:38.809456 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:38.809430    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerStarted","Data":"c94ed4b47471a7a20b3c842fc885b2fe947e50599cdcc75700b0650777c756b7"}
Sep 15 11:25:39.658701 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:39.658669    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:39.658945 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:39.658773    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:39.658945 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:39.658781    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4q24x" podUID="125ade64-8f23-480a-906c-4400b6463813"
Sep 15 11:25:39.658945 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:39.658780    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:39.658945 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:39.658857    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdwrx" podUID="86729383-b3ac-4559-b996-929287275e55"
Sep 15 11:25:39.658945 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:39.658919    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-75nth" podUID="e2c4e4cb-1aed-4d57-a196-f312af6bfe54"
Sep 15 11:25:39.814816 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:39.814785    2574 generic.go:356] "Generic (PLEG): container finished" podID="acfbfdf0-00c1-4101-a771-797fe7db1917" containerID="c94ed4b47471a7a20b3c842fc885b2fe947e50599cdcc75700b0650777c756b7" exitCode=0
Sep 15 11:25:39.815202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:39.814829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerDied","Data":"c94ed4b47471a7a20b3c842fc885b2fe947e50599cdcc75700b0650777c756b7"}
Sep 15 11:25:40.042648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.042622    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-54.ec2.internal" event="NodeReady"
Sep 15 11:25:40.042794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.042739    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 11:25:40.087099 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.087046    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"]
Sep 15 11:25:40.101014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.100856    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"]
Sep 15 11:25:40.101014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.100958    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.104849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.104369    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 11:25:40.104849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.104443    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8nc4g\""
Sep 15 11:25:40.104849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.104456    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 11:25:40.104849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.104377    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.104849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.104373    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.110849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.110827    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"]
Sep 15 11:25:40.110977 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.110954    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"
Sep 15 11:25:40.113001 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.112980    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.113176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.113158    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.113262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.113163    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8jrgs\""
Sep 15 11:25:40.131833 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.131799    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-r6f5v"]
Sep 15 11:25:40.133271 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.133246    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.135505 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.135459    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-j9rnh\""
Sep 15 11:25:40.135505 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.135489    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.135736 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.135720    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 11:25:40.135807 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.135771    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 11:25:40.135807 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.135799    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.145027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.145007    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4pmlm"]
Sep 15 11:25:40.145131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.145100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.147137 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.147119    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 11:25:40.147268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.147166    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2sd6\""
Sep 15 11:25:40.147338 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.147283    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 11:25:40.161183 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.161163    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"]
Sep 15 11:25:40.161288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.161204    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"]
Sep 15 11:25:40.161288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.161228    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"]
Sep 15 11:25:40.161288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.161243    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r6f5v"]
Sep 15 11:25:40.161288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.161259    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pmlm"]
Sep 15 11:25:40.161423 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.161329    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.163474 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.163411    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.163600 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.163518    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 11:25:40.163600 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.163562    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fs9cl\""
Sep 15 11:25:40.163720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.163654    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.218353 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe774b4c-ce81-41fe-9caa-ef108e82db88-serving-cert\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.218550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218360    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwd4z\" (UniqueName: \"kubernetes.io/projected/feb875f0-4764-40ac-9562-0a624e2a3f73-kube-api-access-qwd4z\") pod \"network-check-source-6f8b768455-6w6m6\" (UID: \"feb875f0-4764-40ac-9562-0a624e2a3f73\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"
Sep 15 11:25:40.218550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218403    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.218550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xm7\" (UniqueName: \"kubernetes.io/projected/fe774b4c-ce81-41fe-9caa-ef108e82db88-kube-api-access-x6xm7\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.218550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbct8\" (UniqueName: \"kubernetes.io/projected/63f1b763-8272-482d-aa51-3dad53ae20d0-kube-api-access-hbct8\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.218550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218506    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63f1b763-8272-482d-aa51-3dad53ae20d0-tmp-dir\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.218831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.218831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e139383e-c857-4fa8-bad5-2d3599a92756-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.218831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218702    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1b763-8272-482d-aa51-3dad53ae20d0-config-volume\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.218831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218780    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j7nd\" (UniqueName: \"kubernetes.io/projected/e139383e-c857-4fa8-bad5-2d3599a92756-kube-api-access-9j7nd\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.218831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.218800    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe774b4c-ce81-41fe-9caa-ef108e82db88-config\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.284936 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.284889    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"]
Sep 15 11:25:40.299331 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.299298    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8czc5"]
Sep 15 11:25:40.299490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.299428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.302410 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.302385    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.302410 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.302402    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 11:25:40.302743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.302720    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 11:25:40.303058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.303026    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xhklf\""
Sep 15 11:25:40.303163 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.303076    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.313441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.313415    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-j9rf4"]
Sep 15 11:25:40.313600 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.313588    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.317958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.317830    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.317958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.317890    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 11:25:40.317958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.317898    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 11:25:40.319126 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e139383e-c857-4fa8-bad5-2d3599a92756-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.319226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1b763-8272-482d-aa51-3dad53ae20d0-config-volume\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.319226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmrgf\" (UniqueName: \"kubernetes.io/projected/b0a6f3ad-dee3-4cfe-8d3c-173165b2725a-kube-api-access-cmrgf\") pod \"ingress-canary-4pmlm\" (UID: \"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a\") " pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.319226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319214    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7nd\" (UniqueName: \"kubernetes.io/projected/e139383e-c857-4fa8-bad5-2d3599a92756-kube-api-access-9j7nd\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.319373 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe774b4c-ce81-41fe-9caa-ef108e82db88-config\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.319373 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319266    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe774b4c-ce81-41fe-9caa-ef108e82db88-serving-cert\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.319373 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwd4z\" (UniqueName: \"kubernetes.io/projected/feb875f0-4764-40ac-9562-0a624e2a3f73-kube-api-access-qwd4z\") pod \"network-check-source-6f8b768455-6w6m6\" (UID: \"feb875f0-4764-40ac-9562-0a624e2a3f73\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"
Sep 15 11:25:40.319373 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.319587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xm7\" (UniqueName: \"kubernetes.io/projected/fe774b4c-ce81-41fe-9caa-ef108e82db88-kube-api-access-x6xm7\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.319587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbct8\" (UniqueName: \"kubernetes.io/projected/63f1b763-8272-482d-aa51-3dad53ae20d0-kube-api-access-hbct8\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.319587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319437    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63f1b763-8272-482d-aa51-3dad53ae20d0-tmp-dir\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.319587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319474    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.319587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.319525    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a6f3ad-dee3-4cfe-8d3c-173165b2725a-cert\") pod \"ingress-canary-4pmlm\" (UID: \"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a\") " pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.321118 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.320713    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e139383e-c857-4fa8-bad5-2d3599a92756-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.321118 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.321099    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:40.321264 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.321172    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls podName:63f1b763-8272-482d-aa51-3dad53ae20d0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:40.821152765 +0000 UTC m=+34.598829002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls") pod "dns-default-r6f5v" (UID: "63f1b763-8272-482d-aa51-3dad53ae20d0") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:40.321264 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.321189    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63f1b763-8272-482d-aa51-3dad53ae20d0-config-volume\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.321264 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.321198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63f1b763-8272-482d-aa51-3dad53ae20d0-tmp-dir\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.321516 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.321290    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:40.321516 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.321347    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls podName:e139383e-c857-4fa8-bad5-2d3599a92756 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:40.821329075 +0000 UTC m=+34.599005323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-595h2" (UID: "e139383e-c857-4fa8-bad5-2d3599a92756") : secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:40.321516 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.321438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe774b4c-ce81-41fe-9caa-ef108e82db88-config\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.321792 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.321660    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-g6hmt\""
Sep 15 11:25:40.322788 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.322767    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.325728 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.323736    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 11:25:40.326863 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.326767    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe774b4c-ce81-41fe-9caa-ef108e82db88-serving-cert\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.337975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.337841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbct8\" (UniqueName: \"kubernetes.io/projected/63f1b763-8272-482d-aa51-3dad53ae20d0-kube-api-access-hbct8\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.337975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.337932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7nd\" (UniqueName: \"kubernetes.io/projected/e139383e-c857-4fa8-bad5-2d3599a92756-kube-api-access-9j7nd\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.337975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.337957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xm7\" (UniqueName: \"kubernetes.io/projected/fe774b4c-ce81-41fe-9caa-ef108e82db88-kube-api-access-x6xm7\") pod \"service-ca-operator-cf7c776b7-g76cs\" (UID: \"fe774b4c-ce81-41fe-9caa-ef108e82db88\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.338159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.338001    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwd4z\" (UniqueName: \"kubernetes.io/projected/feb875f0-4764-40ac-9562-0a624e2a3f73-kube-api-access-qwd4z\") pod \"network-check-source-6f8b768455-6w6m6\" (UID: \"feb875f0-4764-40ac-9562-0a624e2a3f73\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"
Sep 15 11:25:40.346143 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.346117    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-d486bd584-zz2nb"]
Sep 15 11:25:40.346794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.346768    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.350354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.349933    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 11:25:40.350354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.350116    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xfs2n\""
Sep 15 11:25:40.350354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.350117    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 11:25:40.351509 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.351237    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.355617 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.355597    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 11:25:40.361816 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.361792    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.386378 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.386344    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"]
Sep 15 11:25:40.386546 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.386389    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8czc5"]
Sep 15 11:25:40.386546 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.386407    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-j9rf4"]
Sep 15 11:25:40.386546 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.386422    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d486bd584-zz2nb"]
Sep 15 11:25:40.386546 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.386539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.387844 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.387807    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54596c4d75-dpzb4"]
Sep 15 11:25:40.388655 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.388635    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 11:25:40.388775 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.388726    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7klxx\""
Sep 15 11:25:40.389110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.388931    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 11:25:40.389110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.388945    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 11:25:40.389110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.388936    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 11:25:40.389110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.389014    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 11:25:40.389110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.389029    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 11:25:40.399694 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.399677    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.401974 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.401954    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n56tl\""
Sep 15 11:25:40.402186 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.402172    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 11:25:40.402263 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.402173    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 11:25:40.403034 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.403017    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 11:25:40.403427 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.403372    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54596c4d75-dpzb4"]
Sep 15 11:25:40.408958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.408941    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 11:25:40.420125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420059    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"
Sep 15 11:25:40.420327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a6f3ad-dee3-4cfe-8d3c-173165b2725a-cert\") pod \"ingress-canary-4pmlm\" (UID: \"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a\") " pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.420408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:40.420408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2chg\" (UniqueName: \"kubernetes.io/projected/b6982f91-29a9-4be1-aab3-a32da7419a25-kube-api-access-x2chg\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.420536 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16a89553-0aa8-4f53-9a52-2ed4ae18c722-trusted-ca-bundle\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.420536 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16a89553-0aa8-4f53-9a52-2ed4ae18c722-serving-cert\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.420536 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmrgf\" (UniqueName: \"kubernetes.io/projected/b0a6f3ad-dee3-4cfe-8d3c-173165b2725a-kube-api-access-cmrgf\") pod \"ingress-canary-4pmlm\" (UID: \"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a\") " pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.420536 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6982f91-29a9-4be1-aab3-a32da7419a25-trusted-ca\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.420536 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.420520    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a02c4676-331d-453f-ac98-9e153472814c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.420586    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs podName:125ade64-8f23-480a-906c-4400b6463813 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:12.420566456 +0000 UTC m=+66.198242696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs") pod "network-metrics-daemon-4q24x" (UID: "125ade64-8f23-480a-906c-4400b6463813") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6982f91-29a9-4be1-aab3-a32da7419a25-config\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16a89553-0aa8-4f53-9a52-2ed4ae18c722-tmp\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16a89553-0aa8-4f53-9a52-2ed4ae18c722-service-ca-bundle\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:40.420779 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tl8\" (UniqueName: \"kubernetes.io/projected/16a89553-0aa8-4f53-9a52-2ed4ae18c722-kube-api-access-r2tl8\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.421160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a02c4676-331d-453f-ac98-9e153472814c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.421160 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.420807    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:40.421160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984jp\" (UniqueName: \"kubernetes.io/projected/a02c4676-331d-453f-ac98-9e153472814c-kube-api-access-984jp\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.421160 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.420885    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret podName:e2c4e4cb-1aed-4d57-a196-f312af6bfe54 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:12.420866518 +0000 UTC m=+66.198542768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret") pod "global-pull-secret-syncer-75nth" (UID: "e2c4e4cb-1aed-4d57-a196-f312af6bfe54") : object "kube-system"/"original-pull-secret" not registered
Sep 15 11:25:40.421160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6982f91-29a9-4be1-aab3-a32da7419a25-serving-cert\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.421160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.420946    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16a89553-0aa8-4f53-9a52-2ed4ae18c722-snapshots\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.423327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.423286    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0a6f3ad-dee3-4cfe-8d3c-173165b2725a-cert\") pod \"ingress-canary-4pmlm\" (UID: \"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a\") " pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.431874 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.431855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmrgf\" (UniqueName: \"kubernetes.io/projected/b0a6f3ad-dee3-4cfe-8d3c-173165b2725a-kube-api-access-cmrgf\") pod \"ingress-canary-4pmlm\" (UID: \"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a\") " pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.442280 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.442255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"
Sep 15 11:25:40.469410 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.469387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pmlm"
Sep 15 11:25:40.522589 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.522367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16a89553-0aa8-4f53-9a52-2ed4ae18c722-serving-cert\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.523154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.523308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.523334    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-trusted-ca\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.523401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbc29\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-kube-api-access-dbc29\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.523634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6982f91-29a9-4be1-aab3-a32da7419a25-config\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.523686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16a89553-0aa8-4f53-9a52-2ed4ae18c722-service-ca-bundle\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524284    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-image-registry-private-configuration\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a02c4676-331d-453f-ac98-9e153472814c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984jp\" (UniqueName: \"kubernetes.io/projected/a02c4676-331d-453f-ac98-9e153472814c-kube-api-access-984jp\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6982f91-29a9-4be1-aab3-a32da7419a25-serving-cert\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524387    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2chg\" (UniqueName: \"kubernetes.io/projected/b6982f91-29a9-4be1-aab3-a32da7419a25-kube-api-access-x2chg\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-bound-sa-token\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a02c4676-331d-453f-ac98-9e153472814c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-ca-trust-extracted\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.526557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524459    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16a89553-0aa8-4f53-9a52-2ed4ae18c722-trusted-ca-bundle\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-installation-pull-secrets\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-stats-auth\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6982f91-29a9-4be1-aab3-a32da7419a25-trusted-ca\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524557    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16a89553-0aa8-4f53-9a52-2ed4ae18c722-tmp\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tl8\" (UniqueName: \"kubernetes.io/projected/16a89553-0aa8-4f53-9a52-2ed4ae18c722-kube-api-access-r2tl8\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-default-certificate\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16a89553-0aa8-4f53-9a52-2ed4ae18c722-snapshots\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56dd\" (UniqueName: \"kubernetes.io/projected/7123915d-2619-4989-adf4-f56b500cd784-kube-api-access-l56dd\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.524840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-certificates\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.526215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6982f91-29a9-4be1-aab3-a32da7419a25-trusted-ca\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.526445    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a02c4676-331d-453f-ac98-9e153472814c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.526839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16a89553-0aa8-4f53-9a52-2ed4ae18c722-service-ca-bundle\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.527121    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6982f91-29a9-4be1-aab3-a32da7419a25-config\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.527704 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.527473    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16a89553-0aa8-4f53-9a52-2ed4ae18c722-tmp\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.528538 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.527568    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/16a89553-0aa8-4f53-9a52-2ed4ae18c722-serving-cert\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.528538 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.527981    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16a89553-0aa8-4f53-9a52-2ed4ae18c722-trusted-ca-bundle\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.528538 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.528448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/16a89553-0aa8-4f53-9a52-2ed4ae18c722-snapshots\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.531168 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.530250    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bw8\" (UniqueName: \"kubernetes.io/projected/86729383-b3ac-4559-b996-929287275e55-kube-api-access-82bw8\") pod \"network-check-target-fdwrx\" (UID: \"86729383-b3ac-4559-b996-929287275e55\") " pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:40.534808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.533363    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6982f91-29a9-4be1-aab3-a32da7419a25-serving-cert\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.534808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.534456    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a02c4676-331d-453f-ac98-9e153472814c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.537230 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.537110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-984jp\" (UniqueName: \"kubernetes.io/projected/a02c4676-331d-453f-ac98-9e153472814c-kube-api-access-984jp\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9brms\" (UID: \"a02c4676-331d-453f-ac98-9e153472814c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.537230 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.537187    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tl8\" (UniqueName: \"kubernetes.io/projected/16a89553-0aa8-4f53-9a52-2ed4ae18c722-kube-api-access-r2tl8\") pod \"insights-operator-5844b856db-j9rf4\" (UID: \"16a89553-0aa8-4f53-9a52-2ed4ae18c722\") " pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.540574 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.539668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2chg\" (UniqueName: \"kubernetes.io/projected/b6982f91-29a9-4be1-aab3-a32da7419a25-kube-api-access-x2chg\") pod \"console-operator-696884fb9c-8czc5\" (UID: \"b6982f91-29a9-4be1-aab3-a32da7419a25\") " pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.612081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.612049    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"
Sep 15 11:25:40.625007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.624955    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:25:40.626210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-bound-sa-token\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-ca-trust-extracted\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-installation-pull-secrets\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626256    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-stats-auth\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.626298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-default-certificate\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626379    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56dd\" (UniqueName: \"kubernetes.io/projected/7123915d-2619-4989-adf4-f56b500cd784-kube-api-access-l56dd\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-certificates\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626443    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.626462    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.126442356 +0000 UTC m=+34.904118593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : configmap references non-existent config key: service-ca.crt
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626513    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.626535    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 11:25:40.626566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-trusted-ca\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626948 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.626575    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.126563683 +0000 UTC m=+34.904239933 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : secret "router-metrics-certs-default" not found
Sep 15 11:25:40.626948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626566    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbc29\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-kube-api-access-dbc29\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626626    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-image-registry-private-configuration\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.626750    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-ca-trust-extracted\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.626948 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.626905    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:40.626948 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.626920    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54596c4d75-dpzb4: secret "image-registry-tls" not found
Sep 15 11:25:40.627274 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.626981    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls podName:b6f8460d-0ee7-483b-9d7d-be45f8e6aca0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.126958519 +0000 UTC m=+34.904634754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls") pod "image-registry-54596c4d75-dpzb4" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0") : secret "image-registry-tls" not found
Sep 15 11:25:40.627274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.627146    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-certificates\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.628078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.628026    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-trusted-ca\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.629477 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.629431    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-installation-pull-secrets\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.629585 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.629564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-image-registry-private-configuration\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.629752 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.629722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-stats-auth\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.629939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.629913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-default-certificate\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.637927 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.637906    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56dd\" (UniqueName: \"kubernetes.io/projected/7123915d-2619-4989-adf4-f56b500cd784-kube-api-access-l56dd\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:40.638026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.638005    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-bound-sa-token\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.638141 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.638126    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbc29\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-kube-api-access-dbc29\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:40.660188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.660023    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-j9rf4"
Sep 15 11:25:40.736605 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.736570    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs"]
Sep 15 11:25:40.739571 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.739545    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6"]
Sep 15 11:25:40.741403 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:40.741210    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe774b4c_ce81_41fe_9caa_ef108e82db88.slice/crio-dd1a0b196464abd6cc6a395aa878fe097ea0d881f9106ffd7bccf1fc5e5b4d03 WatchSource:0}: Error finding container dd1a0b196464abd6cc6a395aa878fe097ea0d881f9106ffd7bccf1fc5e5b4d03: Status 404 returned error can't find the container with id dd1a0b196464abd6cc6a395aa878fe097ea0d881f9106ffd7bccf1fc5e5b4d03
Sep 15 11:25:40.744731 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:40.744703    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb875f0_4764_40ac_9562_0a624e2a3f73.slice/crio-c75648285b4feb56a8b93414da6e206b1521bf63df0e35c4295a45af09373a25 WatchSource:0}: Error finding container c75648285b4feb56a8b93414da6e206b1521bf63df0e35c4295a45af09373a25: Status 404 returned error can't find the container with id c75648285b4feb56a8b93414da6e206b1521bf63df0e35c4295a45af09373a25
Sep 15 11:25:40.776572 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.776536    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pmlm"]
Sep 15 11:25:40.779516 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:40.779474    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a6f3ad_dee3_4cfe_8d3c_173165b2725a.slice/crio-d6e0d63be5c623962163f28570354c2f08b7654150e191f872c2556897f3d1cf WatchSource:0}: Error finding container d6e0d63be5c623962163f28570354c2f08b7654150e191f872c2556897f3d1cf: Status 404 returned error can't find the container with id d6e0d63be5c623962163f28570354c2f08b7654150e191f872c2556897f3d1cf
Sep 15 11:25:40.818730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.818689    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pmlm" event={"ID":"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a","Type":"ContainerStarted","Data":"d6e0d63be5c623962163f28570354c2f08b7654150e191f872c2556897f3d1cf"}
Sep 15 11:25:40.820058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.820023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs" event={"ID":"fe774b4c-ce81-41fe-9caa-ef108e82db88","Type":"ContainerStarted","Data":"dd1a0b196464abd6cc6a395aa878fe097ea0d881f9106ffd7bccf1fc5e5b4d03"}
Sep 15 11:25:40.821288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.821262    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6" event={"ID":"feb875f0-4764-40ac-9562-0a624e2a3f73","Type":"ContainerStarted","Data":"c75648285b4feb56a8b93414da6e206b1521bf63df0e35c4295a45af09373a25"}
Sep 15 11:25:40.828760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.828732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:40.828909 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.828881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:40.828961 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.828908    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:40.829025 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.828975    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls podName:e139383e-c857-4fa8-bad5-2d3599a92756 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.8289568 +0000 UTC m=+35.606633036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-595h2" (UID: "e139383e-c857-4fa8-bad5-2d3599a92756") : secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:40.829091 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.829036    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:40.829135 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:40.829098    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls podName:63f1b763-8272-482d-aa51-3dad53ae20d0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:41.829080821 +0000 UTC m=+35.606757061 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls") pod "dns-default-r6f5v" (UID: "63f1b763-8272-482d-aa51-3dad53ae20d0") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:40.922220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.920633    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms"]
Sep 15 11:25:40.926679 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:40.926652    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02c4676_331d_453f_ac98_9e153472814c.slice/crio-3bb348a7e43b11a840e9dfdbafa66181287e574611cf5978e25ce7b64286a425 WatchSource:0}: Error finding container 3bb348a7e43b11a840e9dfdbafa66181287e574611cf5978e25ce7b64286a425: Status 404 returned error can't find the container with id 3bb348a7e43b11a840e9dfdbafa66181287e574611cf5978e25ce7b64286a425
Sep 15 11:25:40.942368 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.942341    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8czc5"]
Sep 15 11:25:40.945505 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:40.945462    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6982f91_29a9_4be1_aab3_a32da7419a25.slice/crio-f03ee4f94b38d7a645a78b369f4a49c06822c80f89f01bd0f9530dbc2ac5870f WatchSource:0}: Error finding container f03ee4f94b38d7a645a78b369f4a49c06822c80f89f01bd0f9530dbc2ac5870f: Status 404 returned error can't find the container with id f03ee4f94b38d7a645a78b369f4a49c06822c80f89f01bd0f9530dbc2ac5870f
Sep 15 11:25:40.969976 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:40.969947    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-j9rf4"]
Sep 15 11:25:40.972686 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:40.972657    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a89553_0aa8_4f53_9a52_2ed4ae18c722.slice/crio-e21483cb264e98ef66c1f79509c7cd190ca259339c937b1630c86d0508cb14a2 WatchSource:0}: Error finding container e21483cb264e98ef66c1f79509c7cd190ca259339c937b1630c86d0508cb14a2: Status 404 returned error can't find the container with id e21483cb264e98ef66c1f79509c7cd190ca259339c937b1630c86d0508cb14a2
Sep 15 11:25:41.130627 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.130590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:41.130812 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.130714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:41.130812 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.130743    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:41.130812 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.130760    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:42.130739894 +0000 UTC m=+35.908416150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : configmap references non-existent config key: service-ca.crt
Sep 15 11:25:41.130984 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.130822    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:41.130984 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.130834    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54596c4d75-dpzb4: secret "image-registry-tls" not found
Sep 15 11:25:41.130984 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.130875    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls podName:b6f8460d-0ee7-483b-9d7d-be45f8e6aca0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:42.130862261 +0000 UTC m=+35.908538502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls") pod "image-registry-54596c4d75-dpzb4" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0") : secret "image-registry-tls" not found
Sep 15 11:25:41.130984 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.130894    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 11:25:41.130984 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.130963    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:42.130944288 +0000 UTC m=+35.908620539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : secret "router-metrics-certs-default" not found
Sep 15 11:25:41.661029 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.659461    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:25:41.661029 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.659969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:41.661029 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.660352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:25:41.665955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.664537    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q47md\""
Sep 15 11:25:41.665955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.664749    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 11:25:41.665955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.665016    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvt2n\""
Sep 15 11:25:41.665955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.665277    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 11:25:41.688174 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.687961    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:41.827346 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.827263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" event={"ID":"b6982f91-29a9-4be1-aab3-a32da7419a25","Type":"ContainerStarted","Data":"f03ee4f94b38d7a645a78b369f4a49c06822c80f89f01bd0f9530dbc2ac5870f"}
Sep 15 11:25:41.830101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.830028    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j9rf4" event={"ID":"16a89553-0aa8-4f53-9a52-2ed4ae18c722","Type":"ContainerStarted","Data":"e21483cb264e98ef66c1f79509c7cd190ca259339c937b1630c86d0508cb14a2"}
Sep 15 11:25:41.832306 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.832237    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms" event={"ID":"a02c4676-331d-453f-ac98-9e153472814c","Type":"ContainerStarted","Data":"3bb348a7e43b11a840e9dfdbafa66181287e574611cf5978e25ce7b64286a425"}
Sep 15 11:25:41.838083 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.836899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:41.838083 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:41.836967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:41.838083 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.837704    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:41.838083 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.837732    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:41.838083 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.837767    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls podName:e139383e-c857-4fa8-bad5-2d3599a92756 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:43.837744566 +0000 UTC m=+37.615420808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-595h2" (UID: "e139383e-c857-4fa8-bad5-2d3599a92756") : secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:41.838083 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:41.837801    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls podName:63f1b763-8272-482d-aa51-3dad53ae20d0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:43.837782896 +0000 UTC m=+37.615459134 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls") pod "dns-default-r6f5v" (UID: "63f1b763-8272-482d-aa51-3dad53ae20d0") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:42.008006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:42.007245    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fdwrx"]
Sep 15 11:25:42.016392 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:25:42.016355    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86729383_b3ac_4559_b996_929287275e55.slice/crio-83e6bf6507bd58ab4a3e29e8d53a3c4509880c0749127e245398edb805ee1f30 WatchSource:0}: Error finding container 83e6bf6507bd58ab4a3e29e8d53a3c4509880c0749127e245398edb805ee1f30: Status 404 returned error can't find the container with id 83e6bf6507bd58ab4a3e29e8d53a3c4509880c0749127e245398edb805ee1f30
Sep 15 11:25:42.139671 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:42.139627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:42.139833 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:42.139678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:42.139833 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:42.139757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:42.139960 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:42.139933    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:44.139913565 +0000 UTC m=+37.917589813 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : configmap references non-existent config key: service-ca.crt
Sep 15 11:25:42.140036 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:42.140028    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 11:25:42.140083 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:42.140068    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:44.140057001 +0000 UTC m=+37.917733240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : secret "router-metrics-certs-default" not found
Sep 15 11:25:42.140146 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:42.140133    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:42.140146 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:42.140144    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54596c4d75-dpzb4: secret "image-registry-tls" not found
Sep 15 11:25:42.140236 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:42.140174    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls podName:b6f8460d-0ee7-483b-9d7d-be45f8e6aca0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:44.140164078 +0000 UTC m=+37.917840315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls") pod "image-registry-54596c4d75-dpzb4" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0") : secret "image-registry-tls" not found
Sep 15 11:25:42.857054 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:42.857017    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fdwrx" event={"ID":"86729383-b3ac-4559-b996-929287275e55","Type":"ContainerStarted","Data":"83e6bf6507bd58ab4a3e29e8d53a3c4509880c0749127e245398edb805ee1f30"}
Sep 15 11:25:43.861315 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:43.861239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:43.861756 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:43.861324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:43.861756 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:43.861381    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:43.861756 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:43.861447    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:43.861756 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:43.861456    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls podName:63f1b763-8272-482d-aa51-3dad53ae20d0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:47.86143402 +0000 UTC m=+41.639110266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls") pod "dns-default-r6f5v" (UID: "63f1b763-8272-482d-aa51-3dad53ae20d0") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:43.861756 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:43.861605    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls podName:e139383e-c857-4fa8-bad5-2d3599a92756 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:47.861584291 +0000 UTC m=+41.639260534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-595h2" (UID: "e139383e-c857-4fa8-bad5-2d3599a92756") : secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:44.163607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:44.163455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:44.163607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:44.163599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:44.163835 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:44.163649    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:48.163624957 +0000 UTC m=+41.941301193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : configmap references non-existent config key: service-ca.crt
Sep 15 11:25:44.163835 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:44.163669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:44.163835 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:44.163725    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 11:25:44.163835 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:44.163774    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:48.163762429 +0000 UTC m=+41.941438688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : secret "router-metrics-certs-default" not found
Sep 15 11:25:44.163835 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:44.163803    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:44.163835 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:44.163821    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54596c4d75-dpzb4: secret "image-registry-tls" not found
Sep 15 11:25:44.164139 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:44.163859    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls podName:b6f8460d-0ee7-483b-9d7d-be45f8e6aca0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:48.163848609 +0000 UTC m=+41.941524843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls") pod "image-registry-54596c4d75-dpzb4" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0") : secret "image-registry-tls" not found
Sep 15 11:25:47.898771 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:47.898729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:47.899198 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:47.898883    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:47.899198 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:47.898880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:47.899198 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:47.898941    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:47.899198 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:47.898948    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls podName:e139383e-c857-4fa8-bad5-2d3599a92756 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:55.898929622 +0000 UTC m=+49.676605870 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-595h2" (UID: "e139383e-c857-4fa8-bad5-2d3599a92756") : secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:47.899198 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:47.899033    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls podName:63f1b763-8272-482d-aa51-3dad53ae20d0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:55.899012193 +0000 UTC m=+49.676688441 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls") pod "dns-default-r6f5v" (UID: "63f1b763-8272-482d-aa51-3dad53ae20d0") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:48.201915 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:48.201823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:48.201915 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:48.201872    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:48.202140 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:48.201920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:48.202140 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:48.201996    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 11:25:48.202140 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:48.202072    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:56.202050589 +0000 UTC m=+49.979726824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : secret "router-metrics-certs-default" not found
Sep 15 11:25:48.202140 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:48.202090    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:56.202082797 +0000 UTC m=+49.979759031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : configmap references non-existent config key: service-ca.crt
Sep 15 11:25:48.202140 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:48.202101    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:48.202140 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:48.202122    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54596c4d75-dpzb4: secret "image-registry-tls" not found
Sep 15 11:25:48.202355 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:48.202175    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls podName:b6f8460d-0ee7-483b-9d7d-be45f8e6aca0 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:56.202158781 +0000 UTC m=+49.979835021 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls") pod "image-registry-54596c4d75-dpzb4" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0") : secret "image-registry-tls" not found
Sep 15 11:25:50.881867 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.881751    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6" event={"ID":"feb875f0-4764-40ac-9562-0a624e2a3f73","Type":"ContainerStarted","Data":"2f8e36e713235d26e698308719f9eb955ea0c2ff2c3cd8eb26572a1278a8246c"}
Sep 15 11:25:50.883657 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.883611    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fdwrx" event={"ID":"86729383-b3ac-4559-b996-929287275e55","Type":"ContainerStarted","Data":"f9e2a2d0eea62b56cc811787cc6e65a3909eb82e27c86b5de8d7ffcaeafecee2"}
Sep 15 11:25:50.883801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.883677    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:25:50.890480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.886041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs" event={"ID":"fe774b4c-ce81-41fe-9caa-ef108e82db88","Type":"ContainerStarted","Data":"88cba89b722d4ddcf048047a092924ebbc5a36a62ffab41801bbb62f25386578"}
Sep 15 11:25:50.890480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.888142    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/0.log"
Sep 15 11:25:50.890480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.888175    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6982f91-29a9-4be1-aab3-a32da7419a25" containerID="7fecf6715a798a611d67723723fbb4c7e9e041a2558ed97a5bb23ce2ad98e314" exitCode=255
Sep 15 11:25:50.890480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.888358    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" event={"ID":"b6982f91-29a9-4be1-aab3-a32da7419a25","Type":"ContainerDied","Data":"7fecf6715a798a611d67723723fbb4c7e9e041a2558ed97a5bb23ce2ad98e314"}
Sep 15 11:25:50.890480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.888610    2574 scope.go:117] "RemoveContainer" containerID="7fecf6715a798a611d67723723fbb4c7e9e041a2558ed97a5bb23ce2ad98e314"
Sep 15 11:25:50.894471 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.894448    2574 generic.go:356] "Generic (PLEG): container finished" podID="acfbfdf0-00c1-4101-a771-797fe7db1917" containerID="12741fdc461bda3049478db37d0dbd1e7d6e53bbb854caa03f6be760b00e2da7" exitCode=0
Sep 15 11:25:50.894595 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.894545    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerDied","Data":"12741fdc461bda3049478db37d0dbd1e7d6e53bbb854caa03f6be760b00e2da7"}
Sep 15 11:25:50.896447 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.896400    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pmlm" event={"ID":"b0a6f3ad-dee3-4cfe-8d3c-173165b2725a","Type":"ContainerStarted","Data":"3dc2937edd184b0d8902d77d119c2d45c9d739c4a92736d006859c8c32013000"}
Sep 15 11:25:50.898407 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.897931    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6w6m6" podStartSLOduration=1.253254095 podStartE2EDuration="10.897918577s" totalImagesPullingTime="9.644664482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:50.897339322 +0000 UTC m=+44.675015580" watchObservedRunningTime="2026-09-15 11:25:50.897918577 +0000 UTC m=+44.675594834"
Sep 15 11:25:50.899724 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.899332    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j9rf4" event={"ID":"16a89553-0aa8-4f53-9a52-2ed4ae18c722","Type":"ContainerStarted","Data":"790922d9348e3b1a054874eb3ec63dead77fca726ebf7e7404abbf80128cb87b"}
Sep 15 11:25:50.902332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.902308    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms" event={"ID":"a02c4676-331d-453f-ac98-9e153472814c","Type":"ContainerStarted","Data":"e55c85f7dc2d6f689bed6ca8d9a17a3cf1c8eb21bfa3fda68c9d84a09e9f4a5f"}
Sep 15 11:25:50.911735 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.911684    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs" podStartSLOduration=1.26202749 podStartE2EDuration="10.911649191s" totalImagesPullingTime="9.649621701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:50.910880795 +0000 UTC m=+44.688557053" watchObservedRunningTime="2026-09-15 11:25:50.911649191 +0000 UTC m=+44.689325450"
Sep 15 11:25:50.948472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.948415    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4pmlm" podStartSLOduration=1.383903401 podStartE2EDuration="10.948391168s" totalImagesPullingTime="9.564487767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:50.946318762 +0000 UTC m=+44.723995024" watchObservedRunningTime="2026-09-15 11:25:50.948391168 +0000 UTC m=+44.726067425"
Sep 15 11:25:50.962207 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:50.962093    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fdwrx" podStartSLOduration=36.580544324 podStartE2EDuration="44.96207473s" totalImagesPullingTime="8.381530406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:50.960741564 +0000 UTC m=+44.738417822" watchObservedRunningTime="2026-09-15 11:25:50.96207473 +0000 UTC m=+44.739750982"
Sep 15 11:25:51.006998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.006859    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms" podStartSLOduration=1.589373505 podStartE2EDuration="11.006842759s" totalImagesPullingTime="9.417469254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:51.00594605 +0000 UTC m=+44.783622307" watchObservedRunningTime="2026-09-15 11:25:51.006842759 +0000 UTC m=+44.784519366"
Sep 15 11:25:51.025309 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.024249    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-j9rf4" podStartSLOduration=1.652979968 podStartE2EDuration="11.024227842s" totalImagesPullingTime="9.371247874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:51.0233189 +0000 UTC m=+44.800995158" watchObservedRunningTime="2026-09-15 11:25:51.024227842 +0000 UTC m=+44.801904100"
Sep 15 11:25:51.910298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.910266    2574 generic.go:356] "Generic (PLEG): container finished" podID="acfbfdf0-00c1-4101-a771-797fe7db1917" containerID="19ec5e4283d0e664c06c6909f781f45a93e2bec5596cbf254d26f9ce2488035b" exitCode=0
Sep 15 11:25:51.910764 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.910352    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerDied","Data":"19ec5e4283d0e664c06c6909f781f45a93e2bec5596cbf254d26f9ce2488035b"}
Sep 15 11:25:51.912143 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.912124    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/1.log"
Sep 15 11:25:51.912549 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.912534    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/0.log"
Sep 15 11:25:51.912615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.912571    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6982f91-29a9-4be1-aab3-a32da7419a25" containerID="cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a" exitCode=255
Sep 15 11:25:51.912663 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.912643    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" event={"ID":"b6982f91-29a9-4be1-aab3-a32da7419a25","Type":"ContainerDied","Data":"cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a"}
Sep 15 11:25:51.912743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.912721    2574 scope.go:117] "RemoveContainer" containerID="7fecf6715a798a611d67723723fbb4c7e9e041a2558ed97a5bb23ce2ad98e314"
Sep 15 11:25:51.912852 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:51.912828    2574 scope.go:117] "RemoveContainer" containerID="cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a"
Sep 15 11:25:51.913029 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:51.913004    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8czc5_openshift-console-operator(b6982f91-29a9-4be1-aab3-a32da7419a25)\"" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" podUID="b6982f91-29a9-4be1-aab3-a32da7419a25"
Sep 15 11:25:52.846199 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.846168    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9hff6"]
Sep 15 11:25:52.865091 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.865063    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hff6"]
Sep 15 11:25:52.865235 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.865140    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:52.868426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.868405    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 11:25:52.868635 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.868620    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jd6pn\""
Sep 15 11:25:52.868770 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.868752    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 11:25:52.918182 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.918157    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/1.log"
Sep 15 11:25:52.918577 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.918470    2574 scope.go:117] "RemoveContainer" containerID="cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a"
Sep 15 11:25:52.918697 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:52.918673    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8czc5_openshift-console-operator(b6982f91-29a9-4be1-aab3-a32da7419a25)\"" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" podUID="b6982f91-29a9-4be1-aab3-a32da7419a25"
Sep 15 11:25:52.921924 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.921901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" event={"ID":"acfbfdf0-00c1-4101-a771-797fe7db1917","Type":"ContainerStarted","Data":"436ef088913b9fe714b03ea2a58dfb9329d15f337ceaf973a2816a624d6e0a0e"}
Sep 15 11:25:52.940909 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.940883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-crio-socket\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:52.941076 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.941056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-data-volume\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:52.941156 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.941091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:52.941232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.941214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:52.941282 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.941266    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfx4d\" (UniqueName: \"kubernetes.io/projected/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-api-access-xfx4d\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:52.952831 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:52.952789    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mdv4q" podStartSLOduration=9.244534933 podStartE2EDuration="46.952760185s" totalImagesPullingTime="37.708225252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:25:52.95207181 +0000 UTC m=+46.729748069" watchObservedRunningTime="2026-09-15 11:25:52.952760185 +0000 UTC m=+46.730436442"
Sep 15 11:25:53.042233 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042201    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-crio-socket\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.042401 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-data-volume\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.042401 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.042401 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042328    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.042401 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfx4d\" (UniqueName: \"kubernetes.io/projected/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-api-access-xfx4d\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.042618 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.042439    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:53.042618 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042449    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-crio-socket\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.042618 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.042464    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:53.042618 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.042552    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:53.542528837 +0000 UTC m=+47.320205086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:53.042618 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.042574    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:53.542563505 +0000 UTC m=+47.320239745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:53.042618 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.042587    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-data-volume\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.053364 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.053336    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfx4d\" (UniqueName: \"kubernetes.io/projected/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-api-access-xfx4d\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.547434 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.547392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.547634 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:53.547477    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:53.547634 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.547537    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:53.547634 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.547606    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:54.547590103 +0000 UTC m=+48.325266346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:53.547634 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.547608    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:53.547874 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:53.547661    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:54.547645962 +0000 UTC m=+48.325322217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:54.555100 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:54.555053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:54.555556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:54.555254    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:54.555556 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:54.555283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:54.555556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:54.555315    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:56.555296985 +0000 UTC m=+50.332973237 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:54.555556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:54.555388    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:54.555556 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:54.555444    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:25:56.555428406 +0000 UTC m=+50.333104641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:55.418009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:55.417976    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jjhhp_285acd19-2f05-4e65-a29e-f44e7c1f84e6/dns-node-resolver/0.log"
Sep 15 11:25:55.967585 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:55.967551    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:25:55.968092 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:55.967614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:25:55.968092 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:55.967693    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 11:25:55.968092 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:55.967754    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls podName:63f1b763-8272-482d-aa51-3dad53ae20d0 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:11.967738059 +0000 UTC m=+65.745414294 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls") pod "dns-default-r6f5v" (UID: "63f1b763-8272-482d-aa51-3dad53ae20d0") : secret "dns-default-metrics-tls" not found
Sep 15 11:25:55.968092 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:55.967754    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:55.968092 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:55.967816    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls podName:e139383e-c857-4fa8-bad5-2d3599a92756 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:11.967803381 +0000 UTC m=+65.745479616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-595h2" (UID: "e139383e-c857-4fa8-bad5-2d3599a92756") : secret "cluster-monitoring-operator-tls" not found
Sep 15 11:25:56.212161 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.212135    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gwcfc_6c9f072c-14e7-4808-b8a2-6ee04358d01d/node-ca/0.log"
Sep 15 11:25:56.270665 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.270581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:56.270793 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.270681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:25:56.270793 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.270701    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:25:56.270793 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.270747    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:12.270728389 +0000 UTC m=+66.048404624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : configmap references non-existent config key: service-ca.crt
Sep 15 11:25:56.270907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.270796    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 11:25:56.270907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.270810    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54596c4d75-dpzb4: secret "image-registry-tls" not found
Sep 15 11:25:56.270907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.270821    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 11:25:56.270907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.270850    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls podName:b6f8460d-0ee7-483b-9d7d-be45f8e6aca0 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:12.270838884 +0000 UTC m=+66.048515119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls") pod "image-registry-54596c4d75-dpzb4" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0") : secret "image-registry-tls" not found
Sep 15 11:25:56.270907 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.270870    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs podName:7123915d-2619-4989-adf4-f56b500cd784 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:12.270855669 +0000 UTC m=+66.048531918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs") pod "router-default-d486bd584-zz2nb" (UID: "7123915d-2619-4989-adf4-f56b500cd784") : secret "router-metrics-certs-default" not found
Sep 15 11:25:56.572751 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.572666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:56.572894 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.572798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:25:56.572894 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.572836    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:56.572894 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.572894    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:00.57288059 +0000 UTC m=+54.350556824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:25:56.573022 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.572915    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:25:56.573022 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:25:56.572966    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:00.572952262 +0000 UTC m=+54.350628496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : configmap "kube-rbac-proxy" not found
Sep 15 11:25:56.811349 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:56.811323    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pmlm_b0a6f3ad-dee3-4cfe-8d3c-173165b2725a/serve-healthcheck-canary/0.log"
Sep 15 11:25:57.613000 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:25:57.612970    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9brms_a02c4676-331d-453f-ac98-9e153472814c/kube-storage-version-migrator-operator/0.log"
Sep 15 11:26:00.607909 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:00.607868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:00.608301 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:00.607944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:00.608301 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:00.607995    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 11:26:00.608301 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:00.608071    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:08.608054832 +0000 UTC m=+62.385731066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : configmap "kube-rbac-proxy" not found
Sep 15 11:26:00.608301 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:00.608087    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:26:00.608301 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:00.608156    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:08.608139725 +0000 UTC m=+62.385815961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:26:00.625799 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:00.625775    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:26:00.625886 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:00.625821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:26:00.626148 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:00.626135    2574 scope.go:117] "RemoveContainer" containerID="cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a"
Sep 15 11:26:00.626301 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:00.626285    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8czc5_openshift-console-operator(b6982f91-29a9-4be1-aab3-a32da7419a25)\"" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" podUID="b6982f91-29a9-4be1-aab3-a32da7419a25"
Sep 15 11:26:05.811752 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:05.811720    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-89gtx"
Sep 15 11:26:08.673087 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:08.673059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:08.673465 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:08.673116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:08.673465 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:08.673207    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 11:26:08.673465 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:08.673254    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls podName:2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:24.673241609 +0000 UTC m=+78.450917844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9hff6" (UID: "2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36") : secret "insights-runtime-extractor-tls" not found
Sep 15 11:26:08.673601 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:08.673582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:12.001955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.001911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:26:12.002558 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.001970    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:26:12.004356 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.004331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63f1b763-8272-482d-aa51-3dad53ae20d0-metrics-tls\") pod \"dns-default-r6f5v\" (UID: \"63f1b763-8272-482d-aa51-3dad53ae20d0\") " pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:26:12.004488 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.004466    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e139383e-c857-4fa8-bad5-2d3599a92756-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-595h2\" (UID: \"e139383e-c857-4fa8-bad5-2d3599a92756\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:26:12.213609 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.213574    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-8nc4g\""
Sep 15 11:26:12.222340 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.222318    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"
Sep 15 11:26:12.255241 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.255164    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z2sd6\""
Sep 15 11:26:12.263875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.263846    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:26:12.304760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.304435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:12.304760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.304475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:12.304760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.304552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:12.305295 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.305255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7123915d-2619-4989-adf4-f56b500cd784-service-ca-bundle\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:12.306937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.306913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7123915d-2619-4989-adf4-f56b500cd784-metrics-certs\") pod \"router-default-d486bd584-zz2nb\" (UID: \"7123915d-2619-4989-adf4-f56b500cd784\") " pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:12.307052 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.307030    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"image-registry-54596c4d75-dpzb4\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") " pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:12.489232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.489063    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2"]
Sep 15 11:26:12.491653 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:12.491622    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode139383e_c857_4fa8_bad5_2d3599a92756.slice/crio-50f658bf940bb2600506780db78e261fed9c208353b1145ba45f820f74226bf9 WatchSource:0}: Error finding container 50f658bf940bb2600506780db78e261fed9c208353b1145ba45f820f74226bf9: Status 404 returned error can't find the container with id 50f658bf940bb2600506780db78e261fed9c208353b1145ba45f820f74226bf9
Sep 15 11:26:12.498830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.498811    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7klxx\""
Sep 15 11:26:12.506624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.506573    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:26:12.506624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.506611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:26:12.507431 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.507409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:12.508978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.508960    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 11:26:12.509074 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.508996    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 11:26:12.513318 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.513294    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n56tl\""
Sep 15 11:26:12.519029 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.519003    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2c4e4cb-1aed-4d57-a196-f312af6bfe54-original-pull-secret\") pod \"global-pull-secret-syncer-75nth\" (UID: \"e2c4e4cb-1aed-4d57-a196-f312af6bfe54\") " pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:26:12.519107 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.519042    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/125ade64-8f23-480a-906c-4400b6463813-metrics-certs\") pod \"network-metrics-daemon-4q24x\" (UID: \"125ade64-8f23-480a-906c-4400b6463813\") " pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:26:12.521953 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.521935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:12.536178 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.536149    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-r6f5v"]
Sep 15 11:26:12.540251 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:12.540221    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f1b763_8272_482d_aa51_3dad53ae20d0.slice/crio-e0a285e1e7eda1f4fa6bdc3af70e8da765aa3de63cf9e229ede9363836f43aeb WatchSource:0}: Error finding container e0a285e1e7eda1f4fa6bdc3af70e8da765aa3de63cf9e229ede9363836f43aeb: Status 404 returned error can't find the container with id e0a285e1e7eda1f4fa6bdc3af70e8da765aa3de63cf9e229ede9363836f43aeb
Sep 15 11:26:12.580314 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.580090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-75nth"
Sep 15 11:26:12.600076 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.599650    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mvt2n\""
Sep 15 11:26:12.607466 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.607440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4q24x"
Sep 15 11:26:12.663985 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.663026    2574 scope.go:117] "RemoveContainer" containerID="cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a"
Sep 15 11:26:12.782821 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.782793    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-d486bd584-zz2nb"]
Sep 15 11:26:12.818635 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.818610    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54596c4d75-dpzb4"]
Sep 15 11:26:12.837410 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:12.837376    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7123915d_2619_4989_adf4_f56b500cd784.slice/crio-ade5e7938efbe9fe196b8b75c291f25f44b982a22203d791c2142ba0d1399987 WatchSource:0}: Error finding container ade5e7938efbe9fe196b8b75c291f25f44b982a22203d791c2142ba0d1399987: Status 404 returned error can't find the container with id ade5e7938efbe9fe196b8b75c291f25f44b982a22203d791c2142ba0d1399987
Sep 15 11:26:12.837674 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:12.837645    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f8460d_0ee7_483b_9d7d_be45f8e6aca0.slice/crio-03d781acde0e8046db10ca6f84213f235062c01cf229a4ef3e382969a4258750 WatchSource:0}: Error finding container 03d781acde0e8046db10ca6f84213f235062c01cf229a4ef3e382969a4258750: Status 404 returned error can't find the container with id 03d781acde0e8046db10ca6f84213f235062c01cf229a4ef3e382969a4258750
Sep 15 11:26:12.858467 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.858442    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-75nth"]
Sep 15 11:26:12.866396 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:12.866374    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c4e4cb_1aed_4d57_a196_f312af6bfe54.slice/crio-de300bb36d1dfd10ec62f14cb31d263bcae3660ee8c301a74f7cd042881d2f75 WatchSource:0}: Error finding container de300bb36d1dfd10ec62f14cb31d263bcae3660ee8c301a74f7cd042881d2f75: Status 404 returned error can't find the container with id de300bb36d1dfd10ec62f14cb31d263bcae3660ee8c301a74f7cd042881d2f75
Sep 15 11:26:12.895639 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.895613    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4q24x"]
Sep 15 11:26:12.898004 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:12.897978    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125ade64_8f23_480a_906c_4400b6463813.slice/crio-0748c4c127feebcebf4c03f512b53cbe426eb804a09813216d45ef0629178a85 WatchSource:0}: Error finding container 0748c4c127feebcebf4c03f512b53cbe426eb804a09813216d45ef0629178a85: Status 404 returned error can't find the container with id 0748c4c127feebcebf4c03f512b53cbe426eb804a09813216d45ef0629178a85
Sep 15 11:26:12.990949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.990911    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d486bd584-zz2nb" event={"ID":"7123915d-2619-4989-adf4-f56b500cd784","Type":"ContainerStarted","Data":"f2ef67846d4fff786178b44271214a829811583580f4b0daabfe80546398aec7"}
Sep 15 11:26:12.991125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.990971    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-d486bd584-zz2nb" event={"ID":"7123915d-2619-4989-adf4-f56b500cd784","Type":"ContainerStarted","Data":"ade5e7938efbe9fe196b8b75c291f25f44b982a22203d791c2142ba0d1399987"}
Sep 15 11:26:12.992467 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.992395    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q24x" event={"ID":"125ade64-8f23-480a-906c-4400b6463813","Type":"ContainerStarted","Data":"0748c4c127feebcebf4c03f512b53cbe426eb804a09813216d45ef0629178a85"}
Sep 15 11:26:12.993802 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.993770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r6f5v" event={"ID":"63f1b763-8272-482d-aa51-3dad53ae20d0","Type":"ContainerStarted","Data":"e0a285e1e7eda1f4fa6bdc3af70e8da765aa3de63cf9e229ede9363836f43aeb"}
Sep 15 11:26:12.995654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.995633    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:26:12.996152 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.996132    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/1.log"
Sep 15 11:26:12.996237 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.996169    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6982f91-29a9-4be1-aab3-a32da7419a25" containerID="fab2094e98ecffc33f868211b6f07e662260341c0482dd1fea47c76b87d96edf" exitCode=255
Sep 15 11:26:12.996303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.996244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" event={"ID":"b6982f91-29a9-4be1-aab3-a32da7419a25","Type":"ContainerDied","Data":"fab2094e98ecffc33f868211b6f07e662260341c0482dd1fea47c76b87d96edf"}
Sep 15 11:26:12.996347 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.996304    2574 scope.go:117] "RemoveContainer" containerID="cbdf592edfa4dccdd994fddf97362884fc0dbb9d2e4f8b96474f482391a06e1a"
Sep 15 11:26:12.996605 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.996585    2574 scope.go:117] "RemoveContainer" containerID="fab2094e98ecffc33f868211b6f07e662260341c0482dd1fea47c76b87d96edf"
Sep 15 11:26:12.996768 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:12.996749    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-8czc5_openshift-console-operator(b6982f91-29a9-4be1-aab3-a32da7419a25)\"" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" podUID="b6982f91-29a9-4be1-aab3-a32da7419a25"
Sep 15 11:26:12.998732 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.998713    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4" event={"ID":"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0","Type":"ContainerStarted","Data":"c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e"}
Sep 15 11:26:12.998825 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.998743    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4" event={"ID":"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0","Type":"ContainerStarted","Data":"03d781acde0e8046db10ca6f84213f235062c01cf229a4ef3e382969a4258750"}
Sep 15 11:26:12.998825 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.998764    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:12.999958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:12.999931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-75nth" event={"ID":"e2c4e4cb-1aed-4d57-a196-f312af6bfe54","Type":"ContainerStarted","Data":"de300bb36d1dfd10ec62f14cb31d263bcae3660ee8c301a74f7cd042881d2f75"}
Sep 15 11:26:13.001068 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:13.001046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2" event={"ID":"e139383e-c857-4fa8-bad5-2d3599a92756","Type":"ContainerStarted","Data":"50f658bf940bb2600506780db78e261fed9c208353b1145ba45f820f74226bf9"}
Sep 15 11:26:13.012179 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:13.012134    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-d486bd584-zz2nb" podStartSLOduration=33.012116877 podStartE2EDuration="33.012116877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:13.010614251 +0000 UTC m=+66.788290509" watchObservedRunningTime="2026-09-15 11:26:13.012116877 +0000 UTC m=+66.789793136"
Sep 15 11:26:13.035699 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:13.035605    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4" podStartSLOduration=33.035520596 podStartE2EDuration="33.035520596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:13.033679113 +0000 UTC m=+66.811355371" watchObservedRunningTime="2026-09-15 11:26:13.035520596 +0000 UTC m=+66.813196855"
Sep 15 11:26:13.507678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:13.507642    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:13.510888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:13.510863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:14.006726 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:14.006694    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:26:14.007226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:14.007199    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:14.008474 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:14.008451    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-d486bd584-zz2nb"
Sep 15 11:26:15.012458 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:15.012382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2" event={"ID":"e139383e-c857-4fa8-bad5-2d3599a92756","Type":"ContainerStarted","Data":"edc44e90cbacac24e5353ff850a6683d8bff56d3ebe138895f4a23924fb161a5"}
Sep 15 11:26:15.029117 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:15.029054    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-595h2" podStartSLOduration=33.358772483 podStartE2EDuration="35.029031536s" totalImagesPullingTime="1.670259053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:15.028020619 +0000 UTC m=+68.805696870" watchObservedRunningTime="2026-09-15 11:26:15.029031536 +0000 UTC m=+68.806707795"
Sep 15 11:26:15.512813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:15.512778    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54596c4d75-dpzb4"]
Sep 15 11:26:18.025463 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.025425    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r6f5v" event={"ID":"63f1b763-8272-482d-aa51-3dad53ae20d0","Type":"ContainerStarted","Data":"3146c3d3d8359720566fd846bec9170286ba401b2db67bdda739e608fb35312c"}
Sep 15 11:26:18.025962 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.025546    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:26:18.025962 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.025570    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-r6f5v" event={"ID":"63f1b763-8272-482d-aa51-3dad53ae20d0","Type":"ContainerStarted","Data":"4bdfd110b95ac1a78a8b79b8b4d153b21de326725fefd6ee4640827bf48979d9"}
Sep 15 11:26:18.027020 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.026993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-75nth" event={"ID":"e2c4e4cb-1aed-4d57-a196-f312af6bfe54","Type":"ContainerStarted","Data":"13a7fbdffb43ab7705134056e731d35949028bec7a681088a6c77922a79c6e46"}
Sep 15 11:26:18.028909 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.028887    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q24x" event={"ID":"125ade64-8f23-480a-906c-4400b6463813","Type":"ContainerStarted","Data":"a692f527f53d9d5f5bd0ad24033c84dc40ec5fca1d2b85e8a71f3b304cd7282a"}
Sep 15 11:26:18.028992 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.028931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4q24x" event={"ID":"125ade64-8f23-480a-906c-4400b6463813","Type":"ContainerStarted","Data":"1b5d1acf569748ba257e0aeac9b5218ec97cb2a4b07c19bfc2c32e7f071f785e"}
Sep 15 11:26:18.042336 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.042292    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-r6f5v" podStartSLOduration=33.112163826 podStartE2EDuration="38.042276013s" totalImagesPullingTime="4.930112187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:18.040911723 +0000 UTC m=+71.818588017" watchObservedRunningTime="2026-09-15 11:26:18.042276013 +0000 UTC m=+71.819952276"
Sep 15 11:26:18.056031 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.055968    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4q24x" podStartSLOduration=67.48796145 podStartE2EDuration="1m12.05594889s" totalImagesPullingTime="4.56798744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:18.055547772 +0000 UTC m=+71.833224030" watchObservedRunningTime="2026-09-15 11:26:18.05594889 +0000 UTC m=+71.833625149"
Sep 15 11:26:18.068922 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:18.068869    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-75nth" podStartSLOduration=67.468649405 podStartE2EDuration="1m12.068853521s" totalImagesPullingTime="4.600204116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:18.067708469 +0000 UTC m=+71.845384725" watchObservedRunningTime="2026-09-15 11:26:18.068853521 +0000 UTC m=+71.846529793"
Sep 15 11:26:20.626060 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:20.626021    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:26:20.626442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:20.626079    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:26:20.626442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:20.626423    2574 scope.go:117] "RemoveContainer" containerID="fab2094e98ecffc33f868211b6f07e662260341c0482dd1fea47c76b87d96edf"
Sep 15 11:26:20.626667 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:20.626645    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-8czc5_openshift-console-operator(b6982f91-29a9-4be1-aab3-a32da7419a25)\"" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" podUID="b6982f91-29a9-4be1-aab3-a32da7419a25"
Sep 15 11:26:21.916023 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:21.915983    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fdwrx"
Sep 15 11:26:22.802258 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.802227    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"]
Sep 15 11:26:22.833086 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.833059    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"]
Sep 15 11:26:22.833200 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.833147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:22.836276 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.836249    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 11:26:22.836276 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.836272    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8hg76\""
Sep 15 11:26:22.836467 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.836287    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 11:26:22.836467 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:22.836272    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 11:26:23.001678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.001638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.002056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.001721    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.002056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.001745    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.002056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.001763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mgkw\" (UniqueName: \"kubernetes.io/projected/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-kube-api-access-8mgkw\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.102940 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.102851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.102940 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.102910    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.102940 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.102932    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.103226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.102950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mgkw\" (UniqueName: \"kubernetes.io/projected/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-kube-api-access-8mgkw\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.103789 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.103765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.105228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.105205    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.105409 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.105388    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.113403 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.113378    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mgkw\" (UniqueName: \"kubernetes.io/projected/ac8d0cdf-af4a-4b1d-89ec-bbe447566941-kube-api-access-8mgkw\") pod \"prometheus-operator-68bcd7f987-t7vf2\" (UID: \"ac8d0cdf-af4a-4b1d-89ec-bbe447566941\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.142360 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.142334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"
Sep 15 11:26:23.430640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:23.430603    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2"]
Sep 15 11:26:23.433263 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:23.433238    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8d0cdf_af4a_4b1d_89ec_bbe447566941.slice/crio-4272a279e0637aec4bbe175c7069b052040c33624570a7a37644a632b0549c41 WatchSource:0}: Error finding container 4272a279e0637aec4bbe175c7069b052040c33624570a7a37644a632b0549c41: Status 404 returned error can't find the container with id 4272a279e0637aec4bbe175c7069b052040c33624570a7a37644a632b0549c41
Sep 15 11:26:24.050732 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:24.050691    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2" event={"ID":"ac8d0cdf-af4a-4b1d-89ec-bbe447566941","Type":"ContainerStarted","Data":"4272a279e0637aec4bbe175c7069b052040c33624570a7a37644a632b0549c41"}
Sep 15 11:26:24.717759 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:24.717723    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:24.719965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:24.719936    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9hff6\" (UID: \"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36\") " pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:24.976149 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:24.975996    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jd6pn\""
Sep 15 11:26:24.984767 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:24.984747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9hff6"
Sep 15 11:26:25.055579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:25.055530    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2" event={"ID":"ac8d0cdf-af4a-4b1d-89ec-bbe447566941","Type":"ContainerStarted","Data":"a5119cec508b5eab12a8ea435a3750bdb6fee94293e97651027959892ac80a21"}
Sep 15 11:26:25.289812 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:25.289715    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9hff6"]
Sep 15 11:26:25.292177 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:25.292154    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5d0dc1_c426_4c8f_89cc_e4b53f16cc36.slice/crio-8b3e66782332fdde1983581a6663627907597cacd89ba71ed638f8560b8b12db WatchSource:0}: Error finding container 8b3e66782332fdde1983581a6663627907597cacd89ba71ed638f8560b8b12db: Status 404 returned error can't find the container with id 8b3e66782332fdde1983581a6663627907597cacd89ba71ed638f8560b8b12db
Sep 15 11:26:26.065275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:26.065240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2" event={"ID":"ac8d0cdf-af4a-4b1d-89ec-bbe447566941","Type":"ContainerStarted","Data":"090fb9be95e9de4861ec4636d1a9d8482bd844f9a6c5dd3ef842d8ac36856d70"}
Sep 15 11:26:26.066971 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:26.066945    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hff6" event={"ID":"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36","Type":"ContainerStarted","Data":"7bba7822e60449537f2f04e79076275fa6a6071ae4e6329d910170e4691f450c"}
Sep 15 11:26:26.067125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:26.066988    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hff6" event={"ID":"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36","Type":"ContainerStarted","Data":"8b3e66782332fdde1983581a6663627907597cacd89ba71ed638f8560b8b12db"}
Sep 15 11:26:26.083629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:26.083582    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-t7vf2" podStartSLOduration=2.581589837 podStartE2EDuration="4.083565225s" totalImagesPullingTime="1.501975388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:26.082382073 +0000 UTC m=+79.860058330" watchObservedRunningTime="2026-09-15 11:26:26.083565225 +0000 UTC m=+79.861241481"
Sep 15 11:26:27.073459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:27.073422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hff6" event={"ID":"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36","Type":"ContainerStarted","Data":"3224e69296915112bf8bc4b105ae61179284e3148c1caaf714d005c787f7908a"}
Sep 15 11:26:28.034066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.034032    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-r6f5v"
Sep 15 11:26:28.079284 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.079244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9hff6" event={"ID":"2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36","Type":"ContainerStarted","Data":"1c400e42c48481089a4efb4b57d40b5ea34f9b7d4dd0c0b630312e405e956b80"}
Sep 15 11:26:28.102647 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.102557    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9hff6" podStartSLOduration=33.794504045 podStartE2EDuration="36.102536934s" totalImagesPullingTime="2.308032889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 11:25:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:28.100687904 +0000 UTC m=+81.878364162" watchObservedRunningTime="2026-09-15 11:26:28.102536934 +0000 UTC m=+81.880213192"
Sep 15 11:26:28.198318 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.198281    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"]
Sep 15 11:26:28.200396 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.200379    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.202714 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.202693    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 11:26:28.202836 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.202695    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-kw9w7\""
Sep 15 11:26:28.202927 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.202914    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 11:26:28.219296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.219270    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"]
Sep 15 11:26:28.255544 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.255511    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"]
Sep 15 11:26:28.257868 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.257849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.260301 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.260271    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pzn5p\""
Sep 15 11:26:28.260439 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.260352    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 11:26:28.260528 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.260435    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 11:26:28.260639 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.260617    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 11:26:28.276146 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.276125    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"]
Sep 15 11:26:28.281410 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.281389    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wjl7v"]
Sep 15 11:26:28.283750 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.283735    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.285707 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.285688    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 11:26:28.285830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.285812    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hsgvp\""
Sep 15 11:26:28.285975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.285958    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 11:26:28.286195 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.286182    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 11:26:28.345103 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345073    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4d7\" (UniqueName: \"kubernetes.io/projected/0e60bc0e-6693-4110-9277-e2af543c7850-kube-api-access-df4d7\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.345274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.345274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.345274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp5rp\" (UniqueName: \"kubernetes.io/projected/9801c738-55f4-478e-8885-0bead54cda7e-kube-api-access-sp5rp\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.345274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345195    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.345274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.345492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9801c738-55f4-478e-8885-0bead54cda7e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.345492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9801c738-55f4-478e-8885-0bead54cda7e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.345492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.345492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.345427    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e60bc0e-6693-4110-9277-e2af543c7850-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.446624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.446624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.446857 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446656    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-root\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.446857 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-wtmp\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.446857 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9801c738-55f4-478e-8885-0bead54cda7e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.446857 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76e62cfc-5992-4d2f-b7ed-798e336c941c-metrics-client-ca\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.446857 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446847    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9801c738-55f4-478e-8885-0bead54cda7e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446871    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446899    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446920    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e60bc0e-6693-4110-9277-e2af543c7850-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446956    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-sys\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.446982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnsb\" (UniqueName: \"kubernetes.io/projected/76e62cfc-5992-4d2f-b7ed-798e336c941c-kube-api-access-clnsb\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:28.447022    2574 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447064    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4d7\" (UniqueName: \"kubernetes.io/projected/0e60bc0e-6693-4110-9277-e2af543c7850-kube-api-access-df4d7\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.447115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-tls\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.447440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447161    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9801c738-55f4-478e-8885-0bead54cda7e-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.447440 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:28.447171    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-tls podName:0e60bc0e-6693-4110-9277-e2af543c7850 nodeName:}" failed. No retries permitted until 2026-09-15 11:26:28.947148673 +0000 UTC m=+82.724824921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-dx44p" (UID: "0e60bc0e-6693-4110-9277-e2af543c7850") : secret "openshift-state-metrics-tls" not found
Sep 15 11:26:28.447440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447194    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.447440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-textfile\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.447440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.447440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp5rp\" (UniqueName: \"kubernetes.io/projected/9801c738-55f4-478e-8885-0bead54cda7e-kube-api-access-sp5rp\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.447708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447581    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9801c738-55f4-478e-8885-0bead54cda7e-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.447933 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447910    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e60bc0e-6693-4110-9277-e2af543c7850-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.448011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.447941    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.449221 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.449197    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.449583 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.449565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9801c738-55f4-478e-8885-0bead54cda7e-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.449760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.449741    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.455105 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.455087    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4d7\" (UniqueName: \"kubernetes.io/projected/0e60bc0e-6693-4110-9277-e2af543c7850-kube-api-access-df4d7\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.455176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.455138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp5rp\" (UniqueName: \"kubernetes.io/projected/9801c738-55f4-478e-8885-0bead54cda7e-kube-api-access-sp5rp\") pod \"kube-state-metrics-59fbdd77d-9vl5v\" (UID: \"9801c738-55f4-478e-8885-0bead54cda7e\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.548219 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-sys\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnsb\" (UniqueName: \"kubernetes.io/projected/76e62cfc-5992-4d2f-b7ed-798e336c941c-kube-api-access-clnsb\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-tls\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548365    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-textfile\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-sys\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-root\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-wtmp\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548515    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76e62cfc-5992-4d2f-b7ed-798e336c941c-metrics-client-ca\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548555    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-root\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548675    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-wtmp\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.548965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.548722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-textfile\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.549020 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.549004    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-accelerators-collector-config\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.549145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.549127    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76e62cfc-5992-4d2f-b7ed-798e336c941c-metrics-client-ca\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.550587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.550568    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.550680 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.550664    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76e62cfc-5992-4d2f-b7ed-798e336c941c-node-exporter-tls\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.556456 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.556438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnsb\" (UniqueName: \"kubernetes.io/projected/76e62cfc-5992-4d2f-b7ed-798e336c941c-kube-api-access-clnsb\") pod \"node-exporter-wjl7v\" (UID: \"76e62cfc-5992-4d2f-b7ed-798e336c941c\") " pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.567666 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.567649    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"
Sep 15 11:26:28.592179 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.592147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wjl7v"
Sep 15 11:26:28.601617 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:28.601569    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e62cfc_5992_4d2f_b7ed_798e336c941c.slice/crio-11622650fa9199c9d2b4e6d8f1efee5d58fe5fc98f831b5ba976fbbb715be40d WatchSource:0}: Error finding container 11622650fa9199c9d2b4e6d8f1efee5d58fe5fc98f831b5ba976fbbb715be40d: Status 404 returned error can't find the container with id 11622650fa9199c9d2b4e6d8f1efee5d58fe5fc98f831b5ba976fbbb715be40d
Sep 15 11:26:28.836362 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.836277    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v"]
Sep 15 11:26:28.839414 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:28.839383    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9801c738_55f4_478e_8885_0bead54cda7e.slice/crio-e16d8b0fa6f739956ee993e58783fbd04a3b844f1fc586bd955d05e61bfd881a WatchSource:0}: Error finding container e16d8b0fa6f739956ee993e58783fbd04a3b844f1fc586bd955d05e61bfd881a: Status 404 returned error can't find the container with id e16d8b0fa6f739956ee993e58783fbd04a3b844f1fc586bd955d05e61bfd881a
Sep 15 11:26:28.951934 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.951892    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:28.954230 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:28.954210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0e60bc0e-6693-4110-9277-e2af543c7850-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-dx44p\" (UID: \"0e60bc0e-6693-4110-9277-e2af543c7850\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:29.085176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:29.085135    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v" event={"ID":"9801c738-55f4-478e-8885-0bead54cda7e","Type":"ContainerStarted","Data":"e16d8b0fa6f739956ee993e58783fbd04a3b844f1fc586bd955d05e61bfd881a"}
Sep 15 11:26:29.086653 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:29.086579    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjl7v" event={"ID":"76e62cfc-5992-4d2f-b7ed-798e336c941c","Type":"ContainerStarted","Data":"11622650fa9199c9d2b4e6d8f1efee5d58fe5fc98f831b5ba976fbbb715be40d"}
Sep 15 11:26:29.109271 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:29.109151    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"
Sep 15 11:26:29.403024 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:29.402990    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p"]
Sep 15 11:26:29.581175 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:29.581137    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e60bc0e_6693_4110_9277_e2af543c7850.slice/crio-371fbf2fac2d1d2555f3b39bb3e914b36ca6eb6530231e3fec06d7fff62b7e5e WatchSource:0}: Error finding container 371fbf2fac2d1d2555f3b39bb3e914b36ca6eb6530231e3fec06d7fff62b7e5e: Status 404 returned error can't find the container with id 371fbf2fac2d1d2555f3b39bb3e914b36ca6eb6530231e3fec06d7fff62b7e5e
Sep 15 11:26:30.096342 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:30.096254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p" event={"ID":"0e60bc0e-6693-4110-9277-e2af543c7850","Type":"ContainerStarted","Data":"4dda56da7b0e32cbd84d5bb38a4eb5e1d0aff0f3105b107ba0f2615420873163"}
Sep 15 11:26:30.096342 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:30.096323    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p" event={"ID":"0e60bc0e-6693-4110-9277-e2af543c7850","Type":"ContainerStarted","Data":"0ce0c1132164144dafa35c064a7b4e1237c4b039a55e9a16863931cd5603e16a"}
Sep 15 11:26:30.096907 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:30.096355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p" event={"ID":"0e60bc0e-6693-4110-9277-e2af543c7850","Type":"ContainerStarted","Data":"371fbf2fac2d1d2555f3b39bb3e914b36ca6eb6530231e3fec06d7fff62b7e5e"}
Sep 15 11:26:30.098164 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:30.098133    2574 generic.go:356] "Generic (PLEG): container finished" podID="76e62cfc-5992-4d2f-b7ed-798e336c941c" containerID="133a54ca5603c9616092e4965ae02ecbb82229108bc3f75e4b2d41882524d5d7" exitCode=0
Sep 15 11:26:30.098294 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:30.098181    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjl7v" event={"ID":"76e62cfc-5992-4d2f-b7ed-798e336c941c","Type":"ContainerDied","Data":"133a54ca5603c9616092e4965ae02ecbb82229108bc3f75e4b2d41882524d5d7"}
Sep 15 11:26:31.105334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.104970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjl7v" event={"ID":"76e62cfc-5992-4d2f-b7ed-798e336c941c","Type":"ContainerStarted","Data":"47da783e98e47fb6082774771f6f4c77dcbc6d16ef793723e766db9a30d737c1"}
Sep 15 11:26:31.105334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.105040    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wjl7v" event={"ID":"76e62cfc-5992-4d2f-b7ed-798e336c941c","Type":"ContainerStarted","Data":"f6ee4e9df5f0f85eb5141dcc92a96b9657d9072bf867f1f4bd77e01007505dc3"}
Sep 15 11:26:31.111105 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.111071    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v" event={"ID":"9801c738-55f4-478e-8885-0bead54cda7e","Type":"ContainerStarted","Data":"c9a2061e23ed3934f92d5c54941738c1902a9060204c57cf02906991a1aef4fe"}
Sep 15 11:26:31.111239 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.111133    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v" event={"ID":"9801c738-55f4-478e-8885-0bead54cda7e","Type":"ContainerStarted","Data":"826f1ae47f3601bd2bc30632997ea819cf68835e64c928ffaec24eb7932e0624"}
Sep 15 11:26:31.111239 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.111161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v" event={"ID":"9801c738-55f4-478e-8885-0bead54cda7e","Type":"ContainerStarted","Data":"10043c0b0f99dd8fcb58b6cbfb43d2a07b8c910c1938c53776329c9f07fdb274"}
Sep 15 11:26:31.113310 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.113282    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p" event={"ID":"0e60bc0e-6693-4110-9277-e2af543c7850","Type":"ContainerStarted","Data":"8e197da35b00de224bd864c11aadfe99fdb244b0342ce72b6e99edf6eb75b1d7"}
Sep 15 11:26:31.126298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.126233    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wjl7v" podStartSLOduration=2.042433055 podStartE2EDuration="3.126210223s" totalImagesPullingTime="1.006248026s" totalInitContainerRuntime="77.529142ms" isStatefulPod=true podCreationTimestamp="2026-09-15 11:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:31.125186143 +0000 UTC m=+84.902862400" watchObservedRunningTime="2026-09-15 11:26:31.126210223 +0000 UTC m=+84.903886483"
Sep 15 11:26:31.152481 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.152430    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-dx44p" podStartSLOduration=2.104031875 podStartE2EDuration="3.152418268s" totalImagesPullingTime="1.048386393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:31.151377526 +0000 UTC m=+84.929053809" watchObservedRunningTime="2026-09-15 11:26:31.152418268 +0000 UTC m=+84.930094524"
Sep 15 11:26:31.175310 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.175252    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-9vl5v" podStartSLOduration=1.877726951 podStartE2EDuration="3.175231462s" totalImagesPullingTime="1.297504511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:31.174916226 +0000 UTC m=+84.952592488" watchObservedRunningTime="2026-09-15 11:26:31.175231462 +0000 UTC m=+84.952907729"
Sep 15 11:26:31.198653 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.198621    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6675878854-twdhs"]
Sep 15 11:26:31.201160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.201140    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.203569 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203524    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2p94q\""
Sep 15 11:26:31.203688 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203567    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 11:26:31.203688 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203578    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 11:26:31.203688 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203605    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 11:26:31.203872 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203855    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 11:26:31.203953 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203935    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-acat0tjqnohs3\""
Sep 15 11:26:31.204008 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.203996    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 11:26:31.214948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.214917    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6675878854-twdhs"]
Sep 15 11:26:31.373748 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-grpc-tls\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373748 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373725    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373762    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fe327d1-6caf-4695-972b-e9a1f0448c9a-metrics-client-ca\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373782    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9kxr\" (UniqueName: \"kubernetes.io/projected/3fe327d1-6caf-4695-972b-e9a1f0448c9a-kube-api-access-b9kxr\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-tls\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.373937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.373928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475053 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475053 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-tls\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475302 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475302 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-grpc-tls\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475302 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475279    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475449 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fe327d1-6caf-4695-972b-e9a1f0448c9a-metrics-client-ca\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.475542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.475489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9kxr\" (UniqueName: \"kubernetes.io/projected/3fe327d1-6caf-4695-972b-e9a1f0448c9a-kube-api-access-b9kxr\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.476176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.476148    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fe327d1-6caf-4695-972b-e9a1f0448c9a-metrics-client-ca\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.477978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.477925    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.477978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.477930    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-tls\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.478146 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.478019    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.478146 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.478124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-grpc-tls\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.478270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.478248    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.478402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.478387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fe327d1-6caf-4695-972b-e9a1f0448c9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.483233 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.483209    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9kxr\" (UniqueName: \"kubernetes.io/projected/3fe327d1-6caf-4695-972b-e9a1f0448c9a-kube-api-access-b9kxr\") pod \"thanos-querier-6675878854-twdhs\" (UID: \"3fe327d1-6caf-4695-972b-e9a1f0448c9a\") " pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.510560 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.510531    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:31.784549 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:31.784515    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6675878854-twdhs"]
Sep 15 11:26:31.787224 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:31.787201    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe327d1_6caf_4695_972b_e9a1f0448c9a.slice/crio-e4ec71471bba1134ca8eacad7416c8535036088436a6089c6169456f1defa6e4 WatchSource:0}: Error finding container e4ec71471bba1134ca8eacad7416c8535036088436a6089c6169456f1defa6e4: Status 404 returned error can't find the container with id e4ec71471bba1134ca8eacad7416c8535036088436a6089c6169456f1defa6e4
Sep 15 11:26:32.118820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:32.118731    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"e4ec71471bba1134ca8eacad7416c8535036088436a6089c6169456f1defa6e4"}
Sep 15 11:26:33.659009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:33.658988    2574 scope.go:117] "RemoveContainer" containerID="fab2094e98ecffc33f868211b6f07e662260341c0482dd1fea47c76b87d96edf"
Sep 15 11:26:34.128841 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.128815    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:26:34.129021 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.128899    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" event={"ID":"b6982f91-29a9-4be1-aab3-a32da7419a25","Type":"ContainerStarted","Data":"663a3809859fb4a249efdcbda40c6c6bc49bc1a069990769562058697a9b41e5"}
Sep 15 11:26:34.129225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.129175    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:26:34.131325 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.131299    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"b0322103e9d29da5b6fb3862a11716ffff56284fcf7ea04514762a79cceea229"}
Sep 15 11:26:34.131426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.131350    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"d1e766cd98cb5f3ac5867f490efb6dce02e967a43b427dfc77844f9f32698e75"}
Sep 15 11:26:34.131426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.131375    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"d87feed65f0c671f0eb3696be1cb417c8d9e67091c2cac4b2020faded1e3c26a"}
Sep 15 11:26:34.148041 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.147988    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8czc5" podStartSLOduration=44.765750077 podStartE2EDuration="54.147975583s" totalImagesPullingTime="9.382225506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:34.146927289 +0000 UTC m=+87.924603547" watchObservedRunningTime="2026-09-15 11:26:34.147975583 +0000 UTC m=+87.925651839"
Sep 15 11:26:34.357229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.357195    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8czc5"
Sep 15 11:26:34.478696 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.478665    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:26:34.482159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.482134    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.486109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.485913    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m9mkq\""
Sep 15 11:26:34.486109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.485927    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 11:26:34.486109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.485933    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 11:26:34.486109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.485971    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 11:26:34.486109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.485991    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 11:26:34.486523 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486248    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 11:26:34.486523 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486385    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 11:26:34.486523 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486518    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 11:26:34.486715 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486609    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 11:26:34.486715 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486651    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 11:26:34.486834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486794    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 11:26:34.487277 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.486996    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c17t5s53g4rha\""
Sep 15 11:26:34.487277 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.487091    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 11:26:34.487277 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.487107    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 11:26:34.489167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.487747    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 11:26:34.489167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.487929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 11:26:34.492950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.492912    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 11:26:34.515040 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.515004    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:26:34.606965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.606847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.606965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.606902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.606965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.606936    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.606965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.606965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.606999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-config\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-web-config\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607079    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb62n\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-kube-api-access-cb62n\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607165    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607194    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607249    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607344    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-config-out\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.607823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.607455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708113 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708092    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb62n\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-kube-api-access-cb62n\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708197    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708745    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-config-out\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708815    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.708950 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708946    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.709339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.709339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.708976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.709339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.709037    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-config\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.709339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.709050    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.709339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.709066    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.709339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.709093    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-web-config\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.710122 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.709790    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.712433 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.710953    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.712433 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.711512    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.712433 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.711808    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.712433 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.712103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.712433 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.712367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.714046 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.714020    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.716236 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.716215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.716949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.716891    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-config-out\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.717867 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.717809    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.717867 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.717813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.718013 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.717911    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-config\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.718357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.718212    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.718357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.718318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-web-config\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.718737 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.718716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.718853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.718832    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.719311 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.719291    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb62n\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-kube-api-access-cb62n\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.719403 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.719352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:34.797532 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:34.797509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:35.096187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.096088    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:26:35.098783 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:26:35.098762    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74c64d3_9c09_4498_bddf_c117f51d9d91.slice/crio-237fe197760bebf174261cb4253f7a0d7362810b9373b4bceafceae15e792526 WatchSource:0}: Error finding container 237fe197760bebf174261cb4253f7a0d7362810b9373b4bceafceae15e792526: Status 404 returned error can't find the container with id 237fe197760bebf174261cb4253f7a0d7362810b9373b4bceafceae15e792526
Sep 15 11:26:35.138195 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.138163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"22f77d9f5d0c6be8b16c73baef9c98758736d47e8f2aa3f1c3f10129d032d2f2"}
Sep 15 11:26:35.138348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.138255    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:35.138348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.138287    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"5acf00e1c2aba971f42477729355678b9d88919b18a0033c2b4174a65c6d671a"}
Sep 15 11:26:35.138348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.138319    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" event={"ID":"3fe327d1-6caf-4695-972b-e9a1f0448c9a","Type":"ContainerStarted","Data":"1e61a00520d273f5182b20ded545eef9e98059642153425738b209a7bd0f170c"}
Sep 15 11:26:35.139381 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.139355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"237fe197760bebf174261cb4253f7a0d7362810b9373b4bceafceae15e792526"}
Sep 15 11:26:35.160832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.160785    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6675878854-twdhs" podStartSLOduration=1.471033016 podStartE2EDuration="4.160765819s" totalImagesPullingTime="2.689732803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:35.159979532 +0000 UTC m=+88.937655790" watchObservedRunningTime="2026-09-15 11:26:35.160765819 +0000 UTC m=+88.938442079"
Sep 15 11:26:35.520747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:35.520709    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:36.146384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:36.146355    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133" exitCode=0
Sep 15 11:26:36.146808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:36.146444    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133"}
Sep 15 11:26:39.168898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:39.167608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96"}
Sep 15 11:26:39.168898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:39.167695    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79"}
Sep 15 11:26:39.168898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:39.167737    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a"}
Sep 15 11:26:40.174922 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.174888    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3"}
Sep 15 11:26:40.175296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.174977    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba"}
Sep 15 11:26:40.175296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.175004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerStarted","Data":"c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1"}
Sep 15 11:26:40.209432 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.209375    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.495263883 podStartE2EDuration="6.209358638s" totalImagesPullingTime="3.691562731s" totalInitContainerRuntime="22.532024ms" isStatefulPod=false podCreationTimestamp="2026-09-15 11:26:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:26:40.206881394 +0000 UTC m=+93.984557652" watchObservedRunningTime="2026-09-15 11:26:40.209358638 +0000 UTC m=+93.987034894"
Sep 15 11:26:40.540069 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.539952    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4" podUID="b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" containerName="registry" containerID="cri-o://c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e" gracePeriod=30
Sep 15 11:26:40.776179 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.776154    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:40.860675 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860596    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-bound-sa-token\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860675 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-trusted-ca\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860675 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860657    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-certificates\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860690    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-image-registry-private-configuration\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860724    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbc29\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-kube-api-access-dbc29\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860745    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860799    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-ca-trust-extracted\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.860917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.860837    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-installation-pull-secrets\") pod \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\" (UID: \"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0\") "
Sep 15 11:26:40.861165 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.861067    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:26:40.861165 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.861097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:26:40.863245 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.863208    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:26:40.863382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.863309    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:26:40.863382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.863371    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:26:40.863601 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.863478    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:26:40.863700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.863683    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-kube-api-access-dbc29" (OuterVolumeSpecName: "kube-api-access-dbc29") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "kube-api-access-dbc29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:26:40.869448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.869421    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" (UID: "b6f8460d-0ee7-483b-9d7d-be45f8e6aca0"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:26:40.961686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961653    2574 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-image-registry-private-configuration\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961682    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbc29\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-kube-api-access-dbc29\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961686 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961693    2574 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-tls\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961889 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961703    2574 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-ca-trust-extracted\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961889 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961711    2574 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-installation-pull-secrets\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961889 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961719    2574 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-bound-sa-token\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961889 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961728    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-trusted-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:40.961889 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:40.961736    2574 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0-registry-certificates\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:26:41.152549 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.152522    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6675878854-twdhs"
Sep 15 11:26:41.182301 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.182261    2574 generic.go:356] "Generic (PLEG): container finished" podID="b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" containerID="c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e" exitCode=0
Sep 15 11:26:41.182301 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.182302    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4"
Sep 15 11:26:41.182847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.182341    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4" event={"ID":"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0","Type":"ContainerDied","Data":"c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e"}
Sep 15 11:26:41.182847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.182403    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54596c4d75-dpzb4" event={"ID":"b6f8460d-0ee7-483b-9d7d-be45f8e6aca0","Type":"ContainerDied","Data":"03d781acde0e8046db10ca6f84213f235062c01cf229a4ef3e382969a4258750"}
Sep 15 11:26:41.182847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.182434    2574 scope.go:117] "RemoveContainer" containerID="c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e"
Sep 15 11:26:41.193897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.193868    2574 scope.go:117] "RemoveContainer" containerID="c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e"
Sep 15 11:26:41.194235 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:26:41.194212    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e\": container with ID starting with c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e not found: ID does not exist" containerID="c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e"
Sep 15 11:26:41.194306 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.194245    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e"} err="failed to get container status \"c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e\": rpc error: code = NotFound desc = could not find container \"c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e\": container with ID starting with c06f7dcbf64e2199e8117f700410b27e98f7e6369e0537c04d86cb07b47d041e not found: ID does not exist"
Sep 15 11:26:41.209375 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.209342    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54596c4d75-dpzb4"]
Sep 15 11:26:41.212456 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:41.212429    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54596c4d75-dpzb4"]
Sep 15 11:26:42.664517 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:42.664465    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" path="/var/lib/kubelet/pods/b6f8460d-0ee7-483b-9d7d-be45f8e6aca0/volumes"
Sep 15 11:26:44.797896 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:44.797848    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:26:57.260733 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:57.260698    2574 generic.go:356] "Generic (PLEG): container finished" podID="fe774b4c-ce81-41fe-9caa-ef108e82db88" containerID="88cba89b722d4ddcf048047a092924ebbc5a36a62ffab41801bbb62f25386578" exitCode=0
Sep 15 11:26:57.261113 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:57.260776    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs" event={"ID":"fe774b4c-ce81-41fe-9caa-ef108e82db88","Type":"ContainerDied","Data":"88cba89b722d4ddcf048047a092924ebbc5a36a62ffab41801bbb62f25386578"}
Sep 15 11:26:57.261162 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:57.261125    2574 scope.go:117] "RemoveContainer" containerID="88cba89b722d4ddcf048047a092924ebbc5a36a62ffab41801bbb62f25386578"
Sep 15 11:26:58.266848 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:26:58.266815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-g76cs" event={"ID":"fe774b4c-ce81-41fe-9caa-ef108e82db88","Type":"ContainerStarted","Data":"78cc975f9d93588abe8cd2264b334dbd7fe7042d0337cc447d1c2782fd121b4d"}
Sep 15 11:27:02.284830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:02.284778    2574 generic.go:356] "Generic (PLEG): container finished" podID="a02c4676-331d-453f-ac98-9e153472814c" containerID="e55c85f7dc2d6f689bed6ca8d9a17a3cf1c8eb21bfa3fda68c9d84a09e9f4a5f" exitCode=0
Sep 15 11:27:02.285179 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:02.284850    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms" event={"ID":"a02c4676-331d-453f-ac98-9e153472814c","Type":"ContainerDied","Data":"e55c85f7dc2d6f689bed6ca8d9a17a3cf1c8eb21bfa3fda68c9d84a09e9f4a5f"}
Sep 15 11:27:02.285240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:02.285206    2574 scope.go:117] "RemoveContainer" containerID="e55c85f7dc2d6f689bed6ca8d9a17a3cf1c8eb21bfa3fda68c9d84a09e9f4a5f"
Sep 15 11:27:03.290475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:03.290443    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9brms" event={"ID":"a02c4676-331d-453f-ac98-9e153472814c","Type":"ContainerStarted","Data":"eebdaf766f89f301daa611e2446bd68d81e7faff30acf5a14afafcd16ad2a351"}
Sep 15 11:27:17.344332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:17.344299    2574 generic.go:356] "Generic (PLEG): container finished" podID="16a89553-0aa8-4f53-9a52-2ed4ae18c722" containerID="790922d9348e3b1a054874eb3ec63dead77fca726ebf7e7404abbf80128cb87b" exitCode=0
Sep 15 11:27:17.344752 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:17.344376    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j9rf4" event={"ID":"16a89553-0aa8-4f53-9a52-2ed4ae18c722","Type":"ContainerDied","Data":"790922d9348e3b1a054874eb3ec63dead77fca726ebf7e7404abbf80128cb87b"}
Sep 15 11:27:17.344793 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:17.344752    2574 scope.go:117] "RemoveContainer" containerID="790922d9348e3b1a054874eb3ec63dead77fca726ebf7e7404abbf80128cb87b"
Sep 15 11:27:18.350104 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:18.350069    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-j9rf4" event={"ID":"16a89553-0aa8-4f53-9a52-2ed4ae18c722","Type":"ContainerStarted","Data":"21c86b2d613b503d1f076e96a9712024dd7db5ef58ce73782a4ed4a0490dedcf"}
Sep 15 11:27:34.798226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:34.798184    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:34.814261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:34.814237    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:35.435451 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:35.435423    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:52.868128 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.868090    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:27:52.868721 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.868435    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="prometheus" containerID="cri-o://5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a" gracePeriod=600
Sep 15 11:27:52.868886 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.868856    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-thanos" containerID="cri-o://304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3" gracePeriod=600
Sep 15 11:27:52.870249 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.868990    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy" containerID="cri-o://383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba" gracePeriod=600
Sep 15 11:27:52.870334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.869012    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-web" containerID="cri-o://c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1" gracePeriod=600
Sep 15 11:27:52.870389 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.869027    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="thanos-sidecar" containerID="cri-o://403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96" gracePeriod=600
Sep 15 11:27:52.870438 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:52.869045    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="config-reloader" containerID="cri-o://ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79" gracePeriod=600
Sep 15 11:27:53.497324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497290    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3" exitCode=0
Sep 15 11:27:53.497324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497316    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba" exitCode=0
Sep 15 11:27:53.497324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497322    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96" exitCode=0
Sep 15 11:27:53.497324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497329    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79" exitCode=0
Sep 15 11:27:53.497324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497334    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a" exitCode=0
Sep 15 11:27:53.497654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497371    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3"}
Sep 15 11:27:53.497654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba"}
Sep 15 11:27:53.497654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497462    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96"}
Sep 15 11:27:53.497654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497488    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79"}
Sep 15 11:27:53.497654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:53.497528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a"}
Sep 15 11:27:54.114100 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.114078    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.199771 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199737    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-1\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.199771 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199779    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-metrics-client-certs\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199804    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-tls-assets\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199826    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-web-config\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199858    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-serving-certs-ca-bundle\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199891    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-db\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199917    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-kubelet-serving-ca-bundle\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.199942    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-tls\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200308 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200282    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.200359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200342    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-config\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200406 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200396    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-metrics-client-ca\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200453 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200429    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-trusted-ca-bundle\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200525 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200456    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-thanos-prometheus-http-client-file\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200524    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-kube-rbac-proxy\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200689 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200599    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb62n\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-kube-api-access-cb62n\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200689 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-grpc-tls\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200689 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-config-out\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200840 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200705    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-2\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200840 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200731    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-0\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.200840 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200763    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a74c64d3-9c09-4498-bddf-c117f51d9d91\" (UID: \"a74c64d3-9c09-4498-bddf-c117f51d9d91\") "
Sep 15 11:27:54.201408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.201049    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.202266 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200446    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.202266 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.201920    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:27:54.202266 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.200698    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.202266 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.202211    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.202848 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.202819    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.203316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.203222    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.203316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.203291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.204128 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.204103    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-config" (OuterVolumeSpecName: "config") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.204128 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.204124    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:27:54.204679 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.204644    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.205343 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.205315    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.206469 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.206431    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-kube-api-access-cb62n" (OuterVolumeSpecName: "kube-api-access-cb62n") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "kube-api-access-cb62n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:27:54.207387 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.207349    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:27:54.209759 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.209722    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.209932 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.209755    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.209932 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.209821    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.210065 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.210030    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-config-out" (OuterVolumeSpecName: "config-out") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:27:54.210105 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.210072    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.219061 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.219012    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-web-config" (OuterVolumeSpecName: "web-config") pod "a74c64d3-9c09-4498-bddf-c117f51d9d91" (UID: "a74c64d3-9c09-4498-bddf-c117f51d9d91"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:27:54.302205 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302155    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302205 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302198    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302205 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302210    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302205 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302222    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302231    2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-metrics-client-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302239    2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-tls-assets\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302250    2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-web-config\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302258    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-k8s-db\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302266    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302275    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302283    2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-config\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302302    2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-configmap-metrics-client-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302312    2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a74c64d3-9c09-4498-bddf-c117f51d9d91-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302321    2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302330    2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-kube-rbac-proxy\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302338    2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302346    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb62n\" (UniqueName: \"kubernetes.io/projected/a74c64d3-9c09-4498-bddf-c117f51d9d91-kube-api-access-cb62n\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302355    2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a74c64d3-9c09-4498-bddf-c117f51d9d91-secret-grpc-tls\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.302475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.302362    2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a74c64d3-9c09-4498-bddf-c117f51d9d91-config-out\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:27:54.504836 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.504756    2574 generic.go:356] "Generic (PLEG): container finished" podID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerID="c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1" exitCode=0
Sep 15 11:27:54.504836 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.504833    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.504998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.504838    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1"}
Sep 15 11:27:54.504998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.504901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a74c64d3-9c09-4498-bddf-c117f51d9d91","Type":"ContainerDied","Data":"237fe197760bebf174261cb4253f7a0d7362810b9373b4bceafceae15e792526"}
Sep 15 11:27:54.504998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.504963    2574 scope.go:117] "RemoveContainer" containerID="304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3"
Sep 15 11:27:54.513582 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.513566    2574 scope.go:117] "RemoveContainer" containerID="383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba"
Sep 15 11:27:54.520372 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.520356    2574 scope.go:117] "RemoveContainer" containerID="c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1"
Sep 15 11:27:54.527385 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.527361    2574 scope.go:117] "RemoveContainer" containerID="403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96"
Sep 15 11:27:54.527813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.527791    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:27:54.533305 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.533286    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:27:54.534781 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.534768    2574 scope.go:117] "RemoveContainer" containerID="ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79"
Sep 15 11:27:54.541220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.541205    2574 scope.go:117] "RemoveContainer" containerID="5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a"
Sep 15 11:27:54.548255 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.548237    2574 scope.go:117] "RemoveContainer" containerID="355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133"
Sep 15 11:27:54.554457 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.554442    2574 scope.go:117] "RemoveContainer" containerID="304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3"
Sep 15 11:27:54.554738 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.554721    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3\": container with ID starting with 304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3 not found: ID does not exist" containerID="304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3"
Sep 15 11:27:54.554801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.554744    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3"} err="failed to get container status \"304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3\": rpc error: code = NotFound desc = could not find container \"304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3\": container with ID starting with 304ad144aaac78f3573450e69582a930f1b62a092a8e0ee6134396245d7b80d3 not found: ID does not exist"
Sep 15 11:27:54.554801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.554760    2574 scope.go:117] "RemoveContainer" containerID="383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba"
Sep 15 11:27:54.555004 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.554987    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba\": container with ID starting with 383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba not found: ID does not exist" containerID="383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba"
Sep 15 11:27:54.555042 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555013    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba"} err="failed to get container status \"383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba\": rpc error: code = NotFound desc = could not find container \"383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba\": container with ID starting with 383f37c196452f538d416ce2310239812107f6c7ed8c03d9405a66274f0936ba not found: ID does not exist"
Sep 15 11:27:54.555042 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555030    2574 scope.go:117] "RemoveContainer" containerID="c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1"
Sep 15 11:27:54.555252 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.555235    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1\": container with ID starting with c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1 not found: ID does not exist" containerID="c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1"
Sep 15 11:27:54.555301 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555256    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1"} err="failed to get container status \"c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1\": rpc error: code = NotFound desc = could not find container \"c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1\": container with ID starting with c4d20a146ea6cd1c4c0010995cd9bbc654d686d66be2042da026bf0a6f4f2ba1 not found: ID does not exist"
Sep 15 11:27:54.555301 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555269    2574 scope.go:117] "RemoveContainer" containerID="403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96"
Sep 15 11:27:54.555464 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.555448    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96\": container with ID starting with 403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96 not found: ID does not exist" containerID="403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96"
Sep 15 11:27:54.555516 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555467    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96"} err="failed to get container status \"403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96\": rpc error: code = NotFound desc = could not find container \"403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96\": container with ID starting with 403763c050cc44a5e1b9294b7c458347c7a3f170284fb6ea4d37a4073eb4ac96 not found: ID does not exist"
Sep 15 11:27:54.555516 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555479    2574 scope.go:117] "RemoveContainer" containerID="ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79"
Sep 15 11:27:54.555699 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.555684    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79\": container with ID starting with ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79 not found: ID does not exist" containerID="ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79"
Sep 15 11:27:54.555747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555702    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79"} err="failed to get container status \"ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79\": rpc error: code = NotFound desc = could not find container \"ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79\": container with ID starting with ce3478b59434cbf9f43ae4546b7e8dd4807d119455a14d39e81ee8e8d0529a79 not found: ID does not exist"
Sep 15 11:27:54.555747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555714    2574 scope.go:117] "RemoveContainer" containerID="5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a"
Sep 15 11:27:54.555905 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.555888    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a\": container with ID starting with 5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a not found: ID does not exist" containerID="5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a"
Sep 15 11:27:54.555980 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555908    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a"} err="failed to get container status \"5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a\": rpc error: code = NotFound desc = could not find container \"5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a\": container with ID starting with 5a62e7f0105cff58ffa483b59796766adf8e3ab59836b6acadb7f7da83abae6a not found: ID does not exist"
Sep 15 11:27:54.555980 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.555919    2574 scope.go:117] "RemoveContainer" containerID="355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133"
Sep 15 11:27:54.556296 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:27:54.556270    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133\": container with ID starting with 355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133 not found: ID does not exist" containerID="355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133"
Sep 15 11:27:54.556392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.556298    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133"} err="failed to get container status \"355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133\": rpc error: code = NotFound desc = could not find container \"355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133\": container with ID starting with 355a40bd6b505e85b75fca4c4d24db5526244084a1a9708f29df5cb74d460133 not found: ID does not exist"
Sep 15 11:27:54.565862 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.565841    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:27:54.566120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566109    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="prometheus"
Sep 15 11:27:54.566159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566122    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="prometheus"
Sep 15 11:27:54.566159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566135    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="init-config-reloader"
Sep 15 11:27:54.566159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566140    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="init-config-reloader"
Sep 15 11:27:54.566159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566146    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="config-reloader"
Sep 15 11:27:54.566159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566152    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="config-reloader"
Sep 15 11:27:54.566159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566158    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566163    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566177    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" containerName="registry"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566182    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" containerName="registry"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566190    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="thanos-sidecar"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566195    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="thanos-sidecar"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566201    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-web"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566206    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-web"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566212    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-thanos"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566218    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-thanos"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566264    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566274    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-web"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566280    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="kube-rbac-proxy-thanos"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566286    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="prometheus"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566292    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6f8460d-0ee7-483b-9d7d-be45f8e6aca0" containerName="registry"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566297    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="config-reloader"
Sep 15 11:27:54.566328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.566305    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" containerName="thanos-sidecar"
Sep 15 11:27:54.571391 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.571375    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.573621 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.573602    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-m9mkq\""
Sep 15 11:27:54.573718 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.573651    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 11:27:54.573810 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.573791    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 11:27:54.573896 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.573882    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 11:27:54.574033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574015    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 11:27:54.574252 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574235    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 11:27:54.574340 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574254    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 11:27:54.574340 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574319    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 11:27:54.574838 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574801    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 11:27:54.574937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574921    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 11:27:54.575004 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574973    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 11:27:54.575056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.574925    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 11:27:54.575179 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.575161    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c17t5s53g4rha\""
Sep 15 11:27:54.575572 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.575519    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 11:27:54.575572 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.575531    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 11:27:54.580395 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.580374    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 11:27:54.583747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.583720    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 11:27:54.587804 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.587784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:27:54.663334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.663299    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a74c64d3-9c09-4498-bddf-c117f51d9d91" path="/var/lib/kubelet/pods/a74c64d3-9c09-4498-bddf-c117f51d9d91/volumes"
Sep 15 11:27:54.705888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.705833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.705888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.705893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.705924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.705945    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.705960    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.705981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjs6\" (UniqueName: \"kubernetes.io/projected/d025fcff-e604-47aa-8ea1-11e4bf6451c0-kube-api-access-xzjs6\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706111    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706128    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d025fcff-e604-47aa-8ea1-11e4bf6451c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-config\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706254    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706525 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706272    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d025fcff-e604-47aa-8ea1-11e4bf6451c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706525 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706525 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706337    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.706525 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.706358    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807320 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807224    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807320 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807280    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807552 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807596 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d025fcff-e604-47aa-8ea1-11e4bf6451c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807631 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807782 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-config\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807782 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807718    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807782 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d025fcff-e604-47aa-8ea1-11e4bf6451c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.807964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807963    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.807993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjs6\" (UniqueName: \"kubernetes.io/projected/d025fcff-e604-47aa-8ea1-11e4bf6451c0-kube-api-access-xzjs6\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808527 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808373    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.808595 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.808566    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.809158 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.809130    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.809682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.809659    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.810837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.810534    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.810837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.810555    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-web-config\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.810837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.810636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.810837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.810721    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.811093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.810903    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.811323 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.811299    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d025fcff-e604-47aa-8ea1-11e4bf6451c0-config-out\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.811415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.811297    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.811729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.811702    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.811841 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.811825    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.812940 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.812917    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.813046 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.812998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-config\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.813121 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.813102    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d025fcff-e604-47aa-8ea1-11e4bf6451c0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.813330 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.813317    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d025fcff-e604-47aa-8ea1-11e4bf6451c0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.813366 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.813315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d025fcff-e604-47aa-8ea1-11e4bf6451c0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.815541 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.815526    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjs6\" (UniqueName: \"kubernetes.io/projected/d025fcff-e604-47aa-8ea1-11e4bf6451c0-kube-api-access-xzjs6\") pod \"prometheus-k8s-0\" (UID: \"d025fcff-e604-47aa-8ea1-11e4bf6451c0\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:54.883145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:54.883106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:27:55.162405 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:55.162244    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 11:27:55.164831 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:27:55.164804    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd025fcff_e604_47aa_8ea1_11e4bf6451c0.slice/crio-d9a8add8eaf2dc571e3ff77ed07da672329c2e75ba6bc5c2942eb3ca18f7cbf5 WatchSource:0}: Error finding container d9a8add8eaf2dc571e3ff77ed07da672329c2e75ba6bc5c2942eb3ca18f7cbf5: Status 404 returned error can't find the container with id d9a8add8eaf2dc571e3ff77ed07da672329c2e75ba6bc5c2942eb3ca18f7cbf5
Sep 15 11:27:55.510930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:55.510836    2574 generic.go:356] "Generic (PLEG): container finished" podID="d025fcff-e604-47aa-8ea1-11e4bf6451c0" containerID="25f1084ce9dd662d26aef2b7de94b38e2e47a0e3c01d65e7b24dc40d8d357a89" exitCode=0
Sep 15 11:27:55.510930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:55.510888    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerDied","Data":"25f1084ce9dd662d26aef2b7de94b38e2e47a0e3c01d65e7b24dc40d8d357a89"}
Sep 15 11:27:55.511158 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:55.510953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"d9a8add8eaf2dc571e3ff77ed07da672329c2e75ba6bc5c2942eb3ca18f7cbf5"}
Sep 15 11:27:56.520041 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.519999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"56cc533bc700170cc597b918f6e33d73d14992e1e20a2742bfae615becd5c721"}
Sep 15 11:27:56.520533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.520081    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"3509c1e040fb8f5700a6fb5f268c9fdff86c1b4a3a7187e87f0dafaba7b21fec"}
Sep 15 11:27:56.520533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.520123    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"a8a5491b72d44d2714cce87f7da74586b3f80bcf817fef2b596b9228239b1cf3"}
Sep 15 11:27:56.520533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.520164    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"41d1c6fe11de16ea055942f480d9205de69584e3d51369fdec1188a5423c10c5"}
Sep 15 11:27:56.520533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.520255    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"e98c9b75a05749c6f06b47cd1cf522339e096002416dc058cdf531bc329dc8b3"}
Sep 15 11:27:56.520533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.520296    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d025fcff-e604-47aa-8ea1-11e4bf6451c0","Type":"ContainerStarted","Data":"a9e384536c48b3d0c5ea851150deb6f8b7292eec31669eb3a4b07efffa2952da"}
Sep 15 11:27:56.549815 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:56.549738    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.532257967 podStartE2EDuration="2.549714657s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.45669ms" isStatefulPod=false podCreationTimestamp="2026-09-15 11:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:27:56.548282331 +0000 UTC m=+170.325958588" watchObservedRunningTime="2026-09-15 11:27:56.549714657 +0000 UTC m=+170.327390915"
Sep 15 11:27:59.884147 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:27:59.884059    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:28:54.884015 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:28:54.883965    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:28:54.899342 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:28:54.899320    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:28:55.776156 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:28:55.776128    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 11:30:06.565960 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:30:06.565932    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:30:06.568442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:30:06.568419    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:30:06.570624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:30:06.570600    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:30:06.573202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:30:06.573183    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:30:06.578473 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:30:06.578458    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 11:31:17.223827 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.223743    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"]
Sep 15 11:31:17.227616 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.227592    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.252625 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.252597    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 11:31:17.252880 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.252862    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7mtn4\""
Sep 15 11:31:17.254093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.254072    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 11:31:17.254200 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.254183    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 11:31:17.254294 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.254275    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 11:31:17.303266 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.303228    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"]
Sep 15 11:31:17.317575 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.317542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s6p\" (UniqueName: \"kubernetes.io/projected/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-kube-api-access-27s6p\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.317575 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.317574    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.317753 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.317650    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-manifests\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.317753 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.317713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-webhook-cert\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.317753 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.317730    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-tmp\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.418639 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.418602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s6p\" (UniqueName: \"kubernetes.io/projected/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-kube-api-access-27s6p\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.418639 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.418640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.418879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.418693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-manifests\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.418879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.418746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-webhook-cert\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.418879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.418770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-tmp\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.419110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.419090    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-manifests\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.421257 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.421227    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-tmp\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.421380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.421361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-webhook-cert\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.421460 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.421439    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.431819 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.431785    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s6p\" (UniqueName: \"kubernetes.io/projected/196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f-kube-api-access-27s6p\") pod \"opendatahub-operator-controller-manager-99f4658c8-nmn4h\" (UID: \"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f\") " pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.537027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.536928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:17.824971 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.824901    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"]
Sep 15 11:31:17.828399 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:31:17.828368    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196ae2e9_86c5_4f2c_9f5a_8db1acf8c44f.slice/crio-2cbaf7ff0647c2081070e58a43637006e3940c193fb03da174551c024bacf30a WatchSource:0}: Error finding container 2cbaf7ff0647c2081070e58a43637006e3940c193fb03da174551c024bacf30a: Status 404 returned error can't find the container with id 2cbaf7ff0647c2081070e58a43637006e3940c193fb03da174551c024bacf30a
Sep 15 11:31:17.830285 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:17.830268    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:31:18.324034 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:18.323992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h" event={"ID":"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f","Type":"ContainerStarted","Data":"2cbaf7ff0647c2081070e58a43637006e3940c193fb03da174551c024bacf30a"}
Sep 15 11:31:21.340171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:21.340132    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h" event={"ID":"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f","Type":"ContainerStarted","Data":"c2ac3c6d78503b6417a7549c3b14dead446d294e915d9a278c292e1635e88524"}
Sep 15 11:31:22.346052 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:22.346018    2574 generic.go:356] "Generic (PLEG): container finished" podID="196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f" containerID="c2ac3c6d78503b6417a7549c3b14dead446d294e915d9a278c292e1635e88524" exitCode=0
Sep 15 11:31:22.346549 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:22.346076    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h" event={"ID":"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f","Type":"ContainerDied","Data":"c2ac3c6d78503b6417a7549c3b14dead446d294e915d9a278c292e1635e88524"}
Sep 15 11:31:23.354269 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:23.354229    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h" event={"ID":"196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f","Type":"ContainerStarted","Data":"0532da56e5e369de383eea9830fe2879c8a47c1904ae4b715922d39939d8e15e"}
Sep 15 11:31:23.354743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:23.354338    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:23.378859 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:23.378802    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h" podStartSLOduration=2.1623061630000002 podStartE2EDuration="6.378783682s" totalImagesPullingTime="3.175419738s" totalInitContainerRuntime="1.041057781s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:31:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:31:23.378088642 +0000 UTC m=+377.155764913" watchObservedRunningTime="2026-09-15 11:31:23.378783682 +0000 UTC m=+377.156459938"
Sep 15 11:31:34.361997 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.361960    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-99f4658c8-nmn4h"
Sep 15 11:31:34.683834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.683801    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"]
Sep 15 11:31:34.689989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.689967    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.692875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.692851    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 11:31:34.692875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.692868    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 11:31:34.693231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.692975    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 11:31:34.693360 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.693339    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qwmrt\""
Sep 15 11:31:34.693512 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.693484    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 11:31:34.693695 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.693680    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 11:31:34.700753 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.700720    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"]
Sep 15 11:31:34.775842 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.775804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b4fg\" (UniqueName: \"kubernetes.io/projected/30ba6338-7204-4efb-b2e7-800454edb3ff-kube-api-access-9b4fg\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.776084 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.776056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30ba6338-7204-4efb-b2e7-800454edb3ff-cert\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.776174 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.776103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30ba6338-7204-4efb-b2e7-800454edb3ff-metrics-cert\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.776237 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.776186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30ba6338-7204-4efb-b2e7-800454edb3ff-manager-config\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.877299 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.877265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b4fg\" (UniqueName: \"kubernetes.io/projected/30ba6338-7204-4efb-b2e7-800454edb3ff-kube-api-access-9b4fg\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.877483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.877331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30ba6338-7204-4efb-b2e7-800454edb3ff-cert\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.877483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.877351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30ba6338-7204-4efb-b2e7-800454edb3ff-metrics-cert\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.877483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.877372    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30ba6338-7204-4efb-b2e7-800454edb3ff-manager-config\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.877990 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.877970    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/30ba6338-7204-4efb-b2e7-800454edb3ff-manager-config\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.879736 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.879713    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30ba6338-7204-4efb-b2e7-800454edb3ff-cert\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.879869 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.879851    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/30ba6338-7204-4efb-b2e7-800454edb3ff-metrics-cert\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:34.886151 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:34.886123    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b4fg\" (UniqueName: \"kubernetes.io/projected/30ba6338-7204-4efb-b2e7-800454edb3ff-kube-api-access-9b4fg\") pod \"lws-controller-manager-f6db84f6f-zqzl9\" (UID: \"30ba6338-7204-4efb-b2e7-800454edb3ff\") " pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:35.000332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:35.000237    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:35.271879 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:35.271728    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"]
Sep 15 11:31:35.274422 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:31:35.274392    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ba6338_7204_4efb_b2e7_800454edb3ff.slice/crio-16cc9fe00da7083c40d9aed42c5255931524c79881bebe8666212a5cd43edaee WatchSource:0}: Error finding container 16cc9fe00da7083c40d9aed42c5255931524c79881bebe8666212a5cd43edaee: Status 404 returned error can't find the container with id 16cc9fe00da7083c40d9aed42c5255931524c79881bebe8666212a5cd43edaee
Sep 15 11:31:35.407621 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:35.407587    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9" event={"ID":"30ba6338-7204-4efb-b2e7-800454edb3ff","Type":"ContainerStarted","Data":"16cc9fe00da7083c40d9aed42c5255931524c79881bebe8666212a5cd43edaee"}
Sep 15 11:31:38.143535 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.143478    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"]
Sep 15 11:31:38.146985 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.146961    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.151340 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.151312    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 11:31:38.151582 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.151561    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 11:31:38.152040 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.152024    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9dx2d\""
Sep 15 11:31:38.166441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.166413    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"]
Sep 15 11:31:38.308783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.308741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmltm\" (UniqueName: \"kubernetes.io/projected/dd03552e-6b42-4d24-a27d-1ce40473cffc-kube-api-access-lmltm\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.308991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.308809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd03552e-6b42-4d24-a27d-1ce40473cffc-tmp\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.308991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.308929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03552e-6b42-4d24-a27d-1ce40473cffc-tls-certs\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.409599 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.409516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd03552e-6b42-4d24-a27d-1ce40473cffc-tmp\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.409858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.409608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03552e-6b42-4d24-a27d-1ce40473cffc-tls-certs\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.409858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.409645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmltm\" (UniqueName: \"kubernetes.io/projected/dd03552e-6b42-4d24-a27d-1ce40473cffc-kube-api-access-lmltm\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.411822 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.411794    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dd03552e-6b42-4d24-a27d-1ce40473cffc-tmp\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.411989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.411972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd03552e-6b42-4d24-a27d-1ce40473cffc-tls-certs\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.418567 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.418491    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmltm\" (UniqueName: \"kubernetes.io/projected/dd03552e-6b42-4d24-a27d-1ce40473cffc-kube-api-access-lmltm\") pod \"kube-auth-proxy-59bc95f796-gmfhw\" (UID: \"dd03552e-6b42-4d24-a27d-1ce40473cffc\") " pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.426276 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.426242    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9" event={"ID":"30ba6338-7204-4efb-b2e7-800454edb3ff","Type":"ContainerStarted","Data":"6b28bda00116589eeadae7f1d1b563ec7010848934218e145efbc1211ababdb3"}
Sep 15 11:31:38.426542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.426524    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:31:38.447570 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.447520    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9" podStartSLOduration=2.153348392 podStartE2EDuration="4.447488106s" totalImagesPullingTime="2.294139714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:31:38.446846485 +0000 UTC m=+392.224522743" watchObservedRunningTime="2026-09-15 11:31:38.447488106 +0000 UTC m=+392.225164364"
Sep 15 11:31:38.456913 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.456887    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"
Sep 15 11:31:38.727852 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:38.727788    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw"]
Sep 15 11:31:38.730413 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:31:38.730385    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd03552e_6b42_4d24_a27d_1ce40473cffc.slice/crio-009fafd0eba6e9d41bf5f8c81b5b5256e0df00b52ab1ac4ccea1ef4b04978bdf WatchSource:0}: Error finding container 009fafd0eba6e9d41bf5f8c81b5b5256e0df00b52ab1ac4ccea1ef4b04978bdf: Status 404 returned error can't find the container with id 009fafd0eba6e9d41bf5f8c81b5b5256e0df00b52ab1ac4ccea1ef4b04978bdf
Sep 15 11:31:39.435445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:39.435369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw" event={"ID":"dd03552e-6b42-4d24-a27d-1ce40473cffc","Type":"ContainerStarted","Data":"009fafd0eba6e9d41bf5f8c81b5b5256e0df00b52ab1ac4ccea1ef4b04978bdf"}
Sep 15 11:31:41.446872 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:41.446834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw" event={"ID":"dd03552e-6b42-4d24-a27d-1ce40473cffc","Type":"ContainerStarted","Data":"84d212e4c56ff2145652a8863b18c9598758dd71fb751afeec6281ebaf64ade3"}
Sep 15 11:31:41.466700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:41.466635    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59bc95f796-gmfhw" podStartSLOduration=0.882457297 podStartE2EDuration="3.466616775s" totalImagesPullingTime="2.584159478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:31:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:31:41.466088571 +0000 UTC m=+395.243764829" watchObservedRunningTime="2026-09-15 11:31:41.466616775 +0000 UTC m=+395.244293033"
Sep 15 11:31:49.438612 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:31:49.438573    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f6db84f6f-zqzl9"
Sep 15 11:33:18.162122 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.162084    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"]
Sep 15 11:33:18.164853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.164836    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:18.167897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.167871    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 11:33:18.168014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.167929    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 11:33:18.168669 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.168642    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j9tkn\""
Sep 15 11:33:18.180873 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.180843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xbc9\" (UniqueName: \"kubernetes.io/projected/c547b4f5-6a59-4eb7-9305-19ff46ae5887-kube-api-access-9xbc9\") pod \"limitador-operator-controller-manager-74495c69f8-mjjvc\" (UID: \"c547b4f5-6a59-4eb7-9305-19ff46ae5887\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:18.182291 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.182266    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"]
Sep 15 11:33:18.281409 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.281373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xbc9\" (UniqueName: \"kubernetes.io/projected/c547b4f5-6a59-4eb7-9305-19ff46ae5887-kube-api-access-9xbc9\") pod \"limitador-operator-controller-manager-74495c69f8-mjjvc\" (UID: \"c547b4f5-6a59-4eb7-9305-19ff46ae5887\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:18.293786 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.293757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xbc9\" (UniqueName: \"kubernetes.io/projected/c547b4f5-6a59-4eb7-9305-19ff46ae5887-kube-api-access-9xbc9\") pod \"limitador-operator-controller-manager-74495c69f8-mjjvc\" (UID: \"c547b4f5-6a59-4eb7-9305-19ff46ae5887\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:18.475428 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.475335    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:18.764173 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.764101    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"]
Sep 15 11:33:18.768096 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:33:18.768067    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc547b4f5_6a59_4eb7_9305_19ff46ae5887.slice/crio-1d41fb7db52ff548f1b16e147774240f9f0d748b3ed9cfa4e6beadf4b41d7d9a WatchSource:0}: Error finding container 1d41fb7db52ff548f1b16e147774240f9f0d748b3ed9cfa4e6beadf4b41d7d9a: Status 404 returned error can't find the container with id 1d41fb7db52ff548f1b16e147774240f9f0d748b3ed9cfa4e6beadf4b41d7d9a
Sep 15 11:33:18.880755 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:18.880722    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc" event={"ID":"c547b4f5-6a59-4eb7-9305-19ff46ae5887","Type":"ContainerStarted","Data":"1d41fb7db52ff548f1b16e147774240f9f0d748b3ed9cfa4e6beadf4b41d7d9a"}
Sep 15 11:33:21.897917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:21.897871    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc" event={"ID":"c547b4f5-6a59-4eb7-9305-19ff46ae5887","Type":"ContainerStarted","Data":"215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6"}
Sep 15 11:33:21.898274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:21.897939    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:21.923422 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:21.923373    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc" podStartSLOduration=1.269407716 podStartE2EDuration="3.923357092s" totalImagesPullingTime="2.653949376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:33:21.92255101 +0000 UTC m=+495.700227270" watchObservedRunningTime="2026-09-15 11:33:21.923357092 +0000 UTC m=+495.701033349"
Sep 15 11:33:32.448296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.448258    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"]
Sep 15 11:33:32.448835 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.448440    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" containerName="manager" containerID="cri-o://215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6" gracePeriod=2
Sep 15 11:33:32.450628 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.450604    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:32.457093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.457058    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"]
Sep 15 11:33:32.459302 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:33:32.459270    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mjjvc\" is forbidden: User \"system:node:ip-10-0-142-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-54.ec2.internal' and this object" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:32.489561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.489481    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"]
Sep 15 11:33:32.489937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.489921    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" containerName="manager"
Sep 15 11:33:32.489937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.489936    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" containerName="manager"
Sep 15 11:33:32.490080 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.490005    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" containerName="manager"
Sep 15 11:33:32.492997 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.492978    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:33:32.516669 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.516636    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"]
Sep 15 11:33:32.532521 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:33:32.532460    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mjjvc\" is forbidden: User \"system:node:ip-10-0-142-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-54.ec2.internal' and this object" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:32.617692 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.617663    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tzbj\" (UniqueName: \"kubernetes.io/projected/e2da21ec-b8c8-4daf-b9c5-340991424fe1-kube-api-access-6tzbj\") pod \"limitador-operator-controller-manager-74495c69f8-6gpdr\" (UID: \"e2da21ec-b8c8-4daf-b9c5-340991424fe1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:33:32.680031 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.680007    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:32.718191 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.718127    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tzbj\" (UniqueName: \"kubernetes.io/projected/e2da21ec-b8c8-4daf-b9c5-340991424fe1-kube-api-access-6tzbj\") pod \"limitador-operator-controller-manager-74495c69f8-6gpdr\" (UID: \"e2da21ec-b8c8-4daf-b9c5-340991424fe1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:33:32.733096 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.733062    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tzbj\" (UniqueName: \"kubernetes.io/projected/e2da21ec-b8c8-4daf-b9c5-340991424fe1-kube-api-access-6tzbj\") pod \"limitador-operator-controller-manager-74495c69f8-6gpdr\" (UID: \"e2da21ec-b8c8-4daf-b9c5-340991424fe1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:33:32.818930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.818901    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xbc9\" (UniqueName: \"kubernetes.io/projected/c547b4f5-6a59-4eb7-9305-19ff46ae5887-kube-api-access-9xbc9\") pod \"c547b4f5-6a59-4eb7-9305-19ff46ae5887\" (UID: \"c547b4f5-6a59-4eb7-9305-19ff46ae5887\") "
Sep 15 11:33:32.820874 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.820849    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c547b4f5-6a59-4eb7-9305-19ff46ae5887-kube-api-access-9xbc9" (OuterVolumeSpecName: "kube-api-access-9xbc9") pod "c547b4f5-6a59-4eb7-9305-19ff46ae5887" (UID: "c547b4f5-6a59-4eb7-9305-19ff46ae5887"). InnerVolumeSpecName "kube-api-access-9xbc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:33:32.832009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.831987    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:33:32.920334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.920302    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xbc9\" (UniqueName: \"kubernetes.io/projected/c547b4f5-6a59-4eb7-9305-19ff46ae5887-kube-api-access-9xbc9\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:33:32.948091 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.948063    2574 generic.go:356] "Generic (PLEG): container finished" podID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" containerID="215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6" exitCode=0
Sep 15 11:33:32.948229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.948132    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:32.948229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.948152    2574 scope.go:117] "RemoveContainer" containerID="215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6"
Sep 15 11:33:32.956860 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.956840    2574 scope.go:117] "RemoveContainer" containerID="215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6"
Sep 15 11:33:32.957173 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:33:32.957154    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6\": container with ID starting with 215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6 not found: ID does not exist" containerID="215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6"
Sep 15 11:33:32.957258 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:32.957185    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6"} err="failed to get container status \"215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6\": rpc error: code = NotFound desc = could not find container \"215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6\": container with ID starting with 215bad35b02b10778f0e071bf9e5ee08176525b647fad5c8f2f64a89d6aafad6 not found: ID does not exist"
Sep 15 11:33:32.959115 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:33:32.959092    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mjjvc\" is forbidden: User \"system:node:ip-10-0-142-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-54.ec2.internal' and this object" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:33.105707 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:33.105653    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"]
Sep 15 11:33:33.108256 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:33:33.108230    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2da21ec_b8c8_4daf_b9c5_340991424fe1.slice/crio-8e2f66e33166a8135da125495db2cf5fdb2fae91a289913e2b3f6c5a6f02b562 WatchSource:0}: Error finding container 8e2f66e33166a8135da125495db2cf5fdb2fae91a289913e2b3f6c5a6f02b562: Status 404 returned error can't find the container with id 8e2f66e33166a8135da125495db2cf5fdb2fae91a289913e2b3f6c5a6f02b562
Sep 15 11:33:33.955047 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:33.955013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr" event={"ID":"e2da21ec-b8c8-4daf-b9c5-340991424fe1","Type":"ContainerStarted","Data":"92bea60f1f09befa6e1dca73bbbc13846e3f22d84871fba77e96d12727f4cd85"}
Sep 15 11:33:33.955430 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:33.955068    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr" event={"ID":"e2da21ec-b8c8-4daf-b9c5-340991424fe1","Type":"ContainerStarted","Data":"8e2f66e33166a8135da125495db2cf5fdb2fae91a289913e2b3f6c5a6f02b562"}
Sep 15 11:33:33.955430 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:33.955098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:33:33.973447 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:33.973394    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr" podStartSLOduration=1.9733782560000002 podStartE2EDuration="1.973378256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:33:33.972601155 +0000 UTC m=+507.750277414" watchObservedRunningTime="2026-09-15 11:33:33.973378256 +0000 UTC m=+507.751054512"
Sep 15 11:33:33.974382 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:33:33.974348    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-mjjvc\" is forbidden: User \"system:node:ip-10-0-142-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-142-54.ec2.internal' and this object" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-mjjvc"
Sep 15 11:33:34.664615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:34.664580    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c547b4f5-6a59-4eb7-9305-19ff46ae5887" path="/var/lib/kubelet/pods/c547b4f5-6a59-4eb7-9305-19ff46ae5887/volumes"
Sep 15 11:33:44.963197 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:33:44.963166    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-6gpdr"
Sep 15 11:34:03.223649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.223573    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gqzlb"]
Sep 15 11:34:03.226928 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.226912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:03.229550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.229528    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kj98p\""
Sep 15 11:34:03.245311 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.245283    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gqzlb"]
Sep 15 11:34:03.284858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.284824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkjq\" (UniqueName: \"kubernetes.io/projected/479550be-7504-49c2-a733-2495bf309963-kube-api-access-xtkjq\") pod \"authorino-7bcfd8f9cd-gqzlb\" (UID: \"479550be-7504-49c2-a733-2495bf309963\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:03.385688 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.385653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkjq\" (UniqueName: \"kubernetes.io/projected/479550be-7504-49c2-a733-2495bf309963-kube-api-access-xtkjq\") pod \"authorino-7bcfd8f9cd-gqzlb\" (UID: \"479550be-7504-49c2-a733-2495bf309963\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:03.400237 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.400208    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkjq\" (UniqueName: \"kubernetes.io/projected/479550be-7504-49c2-a733-2495bf309963-kube-api-access-xtkjq\") pod \"authorino-7bcfd8f9cd-gqzlb\" (UID: \"479550be-7504-49c2-a733-2495bf309963\") " pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:03.408367 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.408342    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-424xs"]
Sep 15 11:34:03.411773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.411755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:03.419891 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.419860    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-424xs"]
Sep 15 11:34:03.486210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.486133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czw55\" (UniqueName: \"kubernetes.io/projected/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee-kube-api-access-czw55\") pod \"authorino-59c974bd48-424xs\" (UID: \"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee\") " pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:03.550918 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.550886    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:03.586667 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.586634    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czw55\" (UniqueName: \"kubernetes.io/projected/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee-kube-api-access-czw55\") pod \"authorino-59c974bd48-424xs\" (UID: \"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee\") " pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:03.599934 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.599899    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czw55\" (UniqueName: \"kubernetes.io/projected/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee-kube-api-access-czw55\") pod \"authorino-59c974bd48-424xs\" (UID: \"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee\") " pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:03.721599 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.721566    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:03.818445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.818421    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gqzlb"]
Sep 15 11:34:03.820839 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:34:03.820810    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod479550be_7504_49c2_a733_2495bf309963.slice/crio-bf716d7f5d9118f0e28fb264984ed6445f183bcc8cef95cd9038aadd0f64bdd3 WatchSource:0}: Error finding container bf716d7f5d9118f0e28fb264984ed6445f183bcc8cef95cd9038aadd0f64bdd3: Status 404 returned error can't find the container with id bf716d7f5d9118f0e28fb264984ed6445f183bcc8cef95cd9038aadd0f64bdd3
Sep 15 11:34:03.988036 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:03.987987    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-424xs"]
Sep 15 11:34:03.990486 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:34:03.990461    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabbbe7ed_ab64_4527_bbf7_02ee5dc5c5ee.slice/crio-3759e7e66aa8acf7e1f514e528be396e39fd0a20889ff85b9f1901bf3ff762b5 WatchSource:0}: Error finding container 3759e7e66aa8acf7e1f514e528be396e39fd0a20889ff85b9f1901bf3ff762b5: Status 404 returned error can't find the container with id 3759e7e66aa8acf7e1f514e528be396e39fd0a20889ff85b9f1901bf3ff762b5
Sep 15 11:34:04.091101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:04.091004    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-424xs" event={"ID":"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee","Type":"ContainerStarted","Data":"3759e7e66aa8acf7e1f514e528be396e39fd0a20889ff85b9f1901bf3ff762b5"}
Sep 15 11:34:04.092175 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:04.092150    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb" event={"ID":"479550be-7504-49c2-a733-2495bf309963","Type":"ContainerStarted","Data":"bf716d7f5d9118f0e28fb264984ed6445f183bcc8cef95cd9038aadd0f64bdd3"}
Sep 15 11:34:06.104809 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:06.104767    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-424xs" event={"ID":"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee","Type":"ContainerStarted","Data":"e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448"}
Sep 15 11:34:06.106217 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:06.106193    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb" event={"ID":"479550be-7504-49c2-a733-2495bf309963","Type":"ContainerStarted","Data":"f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e"}
Sep 15 11:34:06.119299 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:06.119217    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-424xs" podStartSLOduration=1.806940321 podStartE2EDuration="3.119203068s" totalImagesPullingTime="1.312262747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:34:06.119027818 +0000 UTC m=+539.896704075" watchObservedRunningTime="2026-09-15 11:34:06.119203068 +0000 UTC m=+539.896879324"
Sep 15 11:34:06.137603 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:06.137535    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb" podStartSLOduration=1.6597032729999999 podStartE2EDuration="3.137514772s" totalImagesPullingTime="1.477811499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:34:06.135537041 +0000 UTC m=+539.913213289" watchObservedRunningTime="2026-09-15 11:34:06.137514772 +0000 UTC m=+539.915191029"
Sep 15 11:34:06.157974 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:06.157945    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gqzlb"]
Sep 15 11:34:08.115191 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:08.115151    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb" podUID="479550be-7504-49c2-a733-2495bf309963" containerName="authorino" containerID="cri-o://f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e" gracePeriod=30
Sep 15 11:34:08.352660 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:08.352638    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:08.428967 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:08.428939    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtkjq\" (UniqueName: \"kubernetes.io/projected/479550be-7504-49c2-a733-2495bf309963-kube-api-access-xtkjq\") pod \"479550be-7504-49c2-a733-2495bf309963\" (UID: \"479550be-7504-49c2-a733-2495bf309963\") "
Sep 15 11:34:08.430955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:08.430928    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479550be-7504-49c2-a733-2495bf309963-kube-api-access-xtkjq" (OuterVolumeSpecName: "kube-api-access-xtkjq") pod "479550be-7504-49c2-a733-2495bf309963" (UID: "479550be-7504-49c2-a733-2495bf309963"). InnerVolumeSpecName "kube-api-access-xtkjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:34:08.529847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:08.529804    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkjq\" (UniqueName: \"kubernetes.io/projected/479550be-7504-49c2-a733-2495bf309963-kube-api-access-xtkjq\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:34:09.121004 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.120968    2574 generic.go:356] "Generic (PLEG): container finished" podID="479550be-7504-49c2-a733-2495bf309963" containerID="f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e" exitCode=0
Sep 15 11:34:09.121442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.121014    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb"
Sep 15 11:34:09.121442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.121060    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb" event={"ID":"479550be-7504-49c2-a733-2495bf309963","Type":"ContainerDied","Data":"f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e"}
Sep 15 11:34:09.121442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.121110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-gqzlb" event={"ID":"479550be-7504-49c2-a733-2495bf309963","Type":"ContainerDied","Data":"bf716d7f5d9118f0e28fb264984ed6445f183bcc8cef95cd9038aadd0f64bdd3"}
Sep 15 11:34:09.121442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.121136    2574 scope.go:117] "RemoveContainer" containerID="f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e"
Sep 15 11:34:09.129104 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.129089    2574 scope.go:117] "RemoveContainer" containerID="f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e"
Sep 15 11:34:09.129335 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:34:09.129319    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e\": container with ID starting with f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e not found: ID does not exist" containerID="f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e"
Sep 15 11:34:09.129384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.129340    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e"} err="failed to get container status \"f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e\": rpc error: code = NotFound desc = could not find container \"f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e\": container with ID starting with f239ff27356ac0b0345e9e37c79eb1ff72609a41ea7ccf856b76d9d275aff57e not found: ID does not exist"
Sep 15 11:34:09.137049 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.137027    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gqzlb"]
Sep 15 11:34:09.140316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:09.140295    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-gqzlb"]
Sep 15 11:34:10.664738 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:10.664708    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="479550be-7504-49c2-a733-2495bf309963" path="/var/lib/kubelet/pods/479550be-7504-49c2-a733-2495bf309963/volumes"
Sep 15 11:34:30.771132 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.771100    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-pvghh"]
Sep 15 11:34:30.771622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.771440    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="479550be-7504-49c2-a733-2495bf309963" containerName="authorino"
Sep 15 11:34:30.771622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.771451    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="479550be-7504-49c2-a733-2495bf309963" containerName="authorino"
Sep 15 11:34:30.771622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.771536    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="479550be-7504-49c2-a733-2495bf309963" containerName="authorino"
Sep 15 11:34:30.776281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.776262    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:30.779198 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.779172    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvghh"]
Sep 15 11:34:30.883568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.883532    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvghh"]
Sep 15 11:34:30.883824 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:34:30.883803    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-995k4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-pvghh" podUID="b9161f5e-d7d3-49c8-a0f8-266657c1b80a"
Sep 15 11:34:30.910466 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.910428    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9j8f2"]
Sep 15 11:34:30.914058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.914036    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:30.916441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.916421    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 11:34:30.921102 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.920961    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9j8f2"]
Sep 15 11:34:30.925209 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:30.925187    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995k4\" (UniqueName: \"kubernetes.io/projected/b9161f5e-d7d3-49c8-a0f8-266657c1b80a-kube-api-access-995k4\") pod \"authorino-557f876f69-pvghh\" (UID: \"b9161f5e-d7d3-49c8-a0f8-266657c1b80a\") " pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:31.009070 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.009044    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9j8f2"]
Sep 15 11:34:31.009358 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:34:31.009339    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-k65gw tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-9j8f2" podUID="9cb15790-18de-4f38-997d-8f597551f36d"
Sep 15 11:34:31.025697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.025630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995k4\" (UniqueName: \"kubernetes.io/projected/b9161f5e-d7d3-49c8-a0f8-266657c1b80a-kube-api-access-995k4\") pod \"authorino-557f876f69-pvghh\" (UID: \"b9161f5e-d7d3-49c8-a0f8-266657c1b80a\") " pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:31.025697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.025679    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb15790-18de-4f38-997d-8f597551f36d-tls-cert\") pod \"authorino-6d4dfc649f-9j8f2\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") " pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.025871 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.025716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k65gw\" (UniqueName: \"kubernetes.io/projected/9cb15790-18de-4f38-997d-8f597551f36d-kube-api-access-k65gw\") pod \"authorino-6d4dfc649f-9j8f2\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") " pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.034691 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.034662    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-995k4\" (UniqueName: \"kubernetes.io/projected/b9161f5e-d7d3-49c8-a0f8-266657c1b80a-kube-api-access-995k4\") pod \"authorino-557f876f69-pvghh\" (UID: \"b9161f5e-d7d3-49c8-a0f8-266657c1b80a\") " pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:31.035862 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.035841    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6647f697-hhp77"]
Sep 15 11:34:31.038713 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.038693    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.046905 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.046882    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6647f697-hhp77"]
Sep 15 11:34:31.127140 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.127105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mccr\" (UniqueName: \"kubernetes.io/projected/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-kube-api-access-4mccr\") pod \"authorino-6f6647f697-hhp77\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") " pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.127320 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.127150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k65gw\" (UniqueName: \"kubernetes.io/projected/9cb15790-18de-4f38-997d-8f597551f36d-kube-api-access-k65gw\") pod \"authorino-6d4dfc649f-9j8f2\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") " pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.127320 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.127234    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-tls-cert\") pod \"authorino-6f6647f697-hhp77\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") " pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.127320 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.127298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb15790-18de-4f38-997d-8f597551f36d-tls-cert\") pod \"authorino-6d4dfc649f-9j8f2\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") " pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.129782 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.129754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb15790-18de-4f38-997d-8f597551f36d-tls-cert\") pod \"authorino-6d4dfc649f-9j8f2\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") " pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.134961 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.134927    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k65gw\" (UniqueName: \"kubernetes.io/projected/9cb15790-18de-4f38-997d-8f597551f36d-kube-api-access-k65gw\") pod \"authorino-6d4dfc649f-9j8f2\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") " pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.222626 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.222597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.222796 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.222598    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:31.227840 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.227814    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mccr\" (UniqueName: \"kubernetes.io/projected/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-kube-api-access-4mccr\") pod \"authorino-6f6647f697-hhp77\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") " pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.227970 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.227882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-tls-cert\") pod \"authorino-6f6647f697-hhp77\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") " pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.228711 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.228690    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:31.230226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.230209    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-tls-cert\") pod \"authorino-6f6647f697-hhp77\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") " pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.231916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.231900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:31.236128 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.236111    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mccr\" (UniqueName: \"kubernetes.io/projected/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-kube-api-access-4mccr\") pod \"authorino-6f6647f697-hhp77\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") " pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.328352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.328269    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k65gw\" (UniqueName: \"kubernetes.io/projected/9cb15790-18de-4f38-997d-8f597551f36d-kube-api-access-k65gw\") pod \"9cb15790-18de-4f38-997d-8f597551f36d\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") "
Sep 15 11:34:31.328352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.328305    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb15790-18de-4f38-997d-8f597551f36d-tls-cert\") pod \"9cb15790-18de-4f38-997d-8f597551f36d\" (UID: \"9cb15790-18de-4f38-997d-8f597551f36d\") "
Sep 15 11:34:31.328611 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.328382    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-995k4\" (UniqueName: \"kubernetes.io/projected/b9161f5e-d7d3-49c8-a0f8-266657c1b80a-kube-api-access-995k4\") pod \"b9161f5e-d7d3-49c8-a0f8-266657c1b80a\" (UID: \"b9161f5e-d7d3-49c8-a0f8-266657c1b80a\") "
Sep 15 11:34:31.330403 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.330375    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cb15790-18de-4f38-997d-8f597551f36d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9cb15790-18de-4f38-997d-8f597551f36d" (UID: "9cb15790-18de-4f38-997d-8f597551f36d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:34:31.330403 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.330377    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb15790-18de-4f38-997d-8f597551f36d-kube-api-access-k65gw" (OuterVolumeSpecName: "kube-api-access-k65gw") pod "9cb15790-18de-4f38-997d-8f597551f36d" (UID: "9cb15790-18de-4f38-997d-8f597551f36d"). InnerVolumeSpecName "kube-api-access-k65gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:34:31.330571 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.330410    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9161f5e-d7d3-49c8-a0f8-266657c1b80a-kube-api-access-995k4" (OuterVolumeSpecName: "kube-api-access-995k4") pod "b9161f5e-d7d3-49c8-a0f8-266657c1b80a" (UID: "b9161f5e-d7d3-49c8-a0f8-266657c1b80a"). InnerVolumeSpecName "kube-api-access-995k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:34:31.351566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.351546    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:34:31.429332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.429303    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k65gw\" (UniqueName: \"kubernetes.io/projected/9cb15790-18de-4f38-997d-8f597551f36d-kube-api-access-k65gw\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:34:31.429332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.429331    2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9cb15790-18de-4f38-997d-8f597551f36d-tls-cert\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:34:31.429490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.429342    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995k4\" (UniqueName: \"kubernetes.io/projected/b9161f5e-d7d3-49c8-a0f8-266657c1b80a-kube-api-access-995k4\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:34:31.632248 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:34:31.632207    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77ee4b3e_77e1_47f4_911f_94d79bd7ea74.slice/crio-08f3cbf78401ec9e19b3bef37db9b4a047b32bc6a26cf7ab02f1114fe6522002 WatchSource:0}: Error finding container 08f3cbf78401ec9e19b3bef37db9b4a047b32bc6a26cf7ab02f1114fe6522002: Status 404 returned error can't find the container with id 08f3cbf78401ec9e19b3bef37db9b4a047b32bc6a26cf7ab02f1114fe6522002
Sep 15 11:34:31.633303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:31.633280    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6647f697-hhp77"]
Sep 15 11:34:32.228771 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.228733    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6647f697-hhp77" event={"ID":"77ee4b3e-77e1-47f4-911f-94d79bd7ea74","Type":"ContainerStarted","Data":"08f3cbf78401ec9e19b3bef37db9b4a047b32bc6a26cf7ab02f1114fe6522002"}
Sep 15 11:34:32.228771 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.228756    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-9j8f2"
Sep 15 11:34:32.229238 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.228756    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-pvghh"
Sep 15 11:34:32.259983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.259953    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvghh"]
Sep 15 11:34:32.262981 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.262960    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-pvghh"]
Sep 15 11:34:32.283657 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.283619    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9j8f2"]
Sep 15 11:34:32.285030 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.285014    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-9j8f2"]
Sep 15 11:34:32.664811 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.664777    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cb15790-18de-4f38-997d-8f597551f36d" path="/var/lib/kubelet/pods/9cb15790-18de-4f38-997d-8f597551f36d/volumes"
Sep 15 11:34:32.665031 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:32.665018    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9161f5e-d7d3-49c8-a0f8-266657c1b80a" path="/var/lib/kubelet/pods/b9161f5e-d7d3-49c8-a0f8-266657c1b80a/volumes"
Sep 15 11:34:33.234699 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.234665    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6647f697-hhp77" event={"ID":"77ee4b3e-77e1-47f4-911f-94d79bd7ea74","Type":"ContainerStarted","Data":"96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6"}
Sep 15 11:34:33.252204 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.252154    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6647f697-hhp77" podStartSLOduration=1.570653855 podStartE2EDuration="2.252134243s" totalImagesPullingTime="681.480388ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:34:33.251608197 +0000 UTC m=+567.029284456" watchObservedRunningTime="2026-09-15 11:34:33.252134243 +0000 UTC m=+567.029810502"
Sep 15 11:34:33.278081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.278052    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-424xs"]
Sep 15 11:34:33.278256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.278233    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-424xs" podUID="abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" containerName="authorino" containerID="cri-o://e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448" gracePeriod=30
Sep 15 11:34:33.513690 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.513669    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:33.647520 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.647446    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czw55\" (UniqueName: \"kubernetes.io/projected/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee-kube-api-access-czw55\") pod \"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee\" (UID: \"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee\") "
Sep 15 11:34:33.649526 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.649488    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee-kube-api-access-czw55" (OuterVolumeSpecName: "kube-api-access-czw55") pod "abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" (UID: "abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee"). InnerVolumeSpecName "kube-api-access-czw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:34:33.748773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:33.748678    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czw55\" (UniqueName: \"kubernetes.io/projected/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee-kube-api-access-czw55\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:34:34.239850 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.239816    2574 generic.go:356] "Generic (PLEG): container finished" podID="abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" containerID="e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448" exitCode=0
Sep 15 11:34:34.240316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.239879    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-424xs"
Sep 15 11:34:34.240316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.239910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-424xs" event={"ID":"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee","Type":"ContainerDied","Data":"e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448"}
Sep 15 11:34:34.240316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.239962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-424xs" event={"ID":"abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee","Type":"ContainerDied","Data":"3759e7e66aa8acf7e1f514e528be396e39fd0a20889ff85b9f1901bf3ff762b5"}
Sep 15 11:34:34.240316 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.239986    2574 scope.go:117] "RemoveContainer" containerID="e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448"
Sep 15 11:34:34.248881 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.248863    2574 scope.go:117] "RemoveContainer" containerID="e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448"
Sep 15 11:34:34.249142 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:34:34.249124    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448\": container with ID starting with e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448 not found: ID does not exist" containerID="e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448"
Sep 15 11:34:34.249190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.249150    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448"} err="failed to get container status \"e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448\": rpc error: code = NotFound desc = could not find container \"e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448\": container with ID starting with e286b6dc34e7a17314591c6a41e2f681fae3bf21d9f5c257ad51b0f4a1da1448 not found: ID does not exist"
Sep 15 11:34:34.261665 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.261638    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-424xs"]
Sep 15 11:34:34.265883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.265863    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-424xs"]
Sep 15 11:34:34.664740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:34:34.664709    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" path="/var/lib/kubelet/pods/abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee/volumes"
Sep 15 11:35:06.597876 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:06.597844    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:35:06.598883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:06.598862    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:35:06.602036 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:06.602009    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:35:06.603055 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:06.603036    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:35:17.569880 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.569847    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"]
Sep 15 11:35:17.570243 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.570199    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" containerName="authorino"
Sep 15 11:35:17.570243 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.570210    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" containerName="authorino"
Sep 15 11:35:17.570312 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.570272    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbbe7ed-ab64-4527-bbf7-02ee5dc5c5ee" containerName="authorino"
Sep 15 11:35:17.574788 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.574767    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.577007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.576986    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5frzm\""
Sep 15 11:35:17.577264 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.577250    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 11:35:17.586425 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.586400    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"]
Sep 15 11:35:17.630078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.630040    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab02359b-112f-4103-bc03-344ca271d9a9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6lkql\" (UID: \"ab02359b-112f-4103-bc03-344ca271d9a9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.630274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.630156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdb97\" (UniqueName: \"kubernetes.io/projected/ab02359b-112f-4103-bc03-344ca271d9a9-kube-api-access-qdb97\") pod \"payload-pre-processing-59fbc47bf8-6lkql\" (UID: \"ab02359b-112f-4103-bc03-344ca271d9a9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.731611 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.731576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab02359b-112f-4103-bc03-344ca271d9a9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6lkql\" (UID: \"ab02359b-112f-4103-bc03-344ca271d9a9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.731808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.731645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdb97\" (UniqueName: \"kubernetes.io/projected/ab02359b-112f-4103-bc03-344ca271d9a9-kube-api-access-qdb97\") pod \"payload-pre-processing-59fbc47bf8-6lkql\" (UID: \"ab02359b-112f-4103-bc03-344ca271d9a9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.732335 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.732308    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab02359b-112f-4103-bc03-344ca271d9a9-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-6lkql\" (UID: \"ab02359b-112f-4103-bc03-344ca271d9a9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.744077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.744045    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdb97\" (UniqueName: \"kubernetes.io/projected/ab02359b-112f-4103-bc03-344ca271d9a9-kube-api-access-qdb97\") pod \"payload-pre-processing-59fbc47bf8-6lkql\" (UID: \"ab02359b-112f-4103-bc03-344ca271d9a9\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:17.886723 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:17.886636    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:18.170469 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:18.170445    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"]
Sep 15 11:35:18.172560 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:35:18.172528    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab02359b_112f_4103_bc03_344ca271d9a9.slice/crio-3f5f2f310de811cb41b5fede03261cba2afe19bf686758355093dc25594b3070 WatchSource:0}: Error finding container 3f5f2f310de811cb41b5fede03261cba2afe19bf686758355093dc25594b3070: Status 404 returned error can't find the container with id 3f5f2f310de811cb41b5fede03261cba2afe19bf686758355093dc25594b3070
Sep 15 11:35:18.439977 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:18.439894    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql" event={"ID":"ab02359b-112f-4103-bc03-344ca271d9a9","Type":"ContainerStarted","Data":"3f5f2f310de811cb41b5fede03261cba2afe19bf686758355093dc25594b3070"}
Sep 15 11:35:20.451100 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:20.451058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql" event={"ID":"ab02359b-112f-4103-bc03-344ca271d9a9","Type":"ContainerStarted","Data":"315fc0040af515c86f6f7c477554a1e2e200081e1d1367efab11157f8beabae2"}
Sep 15 11:35:20.451514 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:20.451185    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:20.467045 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:20.466997    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql" podStartSLOduration=1.3405312600000001 podStartE2EDuration="3.466980816s" totalImagesPullingTime="2.126449556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:35:20.465935975 +0000 UTC m=+614.243612232" watchObservedRunningTime="2026-09-15 11:35:20.466980816 +0000 UTC m=+614.244657073"
Sep 15 11:35:31.458571 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:31.458474    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-6lkql"
Sep 15 11:35:37.180275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.180244    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"]
Sep 15 11:35:37.183743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.183724    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.187002 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.186971    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 11:35:37.187143 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.187008    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 11:35:37.187143 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.187030    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-p4bfm\""
Sep 15 11:35:37.187143 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.187035    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 11:35:37.194234 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.194211    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"]
Sep 15 11:35:37.303622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.303584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.303794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.303638    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf424776-9a5e-4b56-8f19-c8135bfc42d8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.303794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.303701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.303794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.303748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.303794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.303763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.303794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.303787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdvf\" (UniqueName: \"kubernetes.io/projected/cf424776-9a5e-4b56-8f19-c8135bfc42d8-kube-api-access-jjdvf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.404927 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.404882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.404927 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.404924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405182 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.404954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdvf\" (UniqueName: \"kubernetes.io/projected/cf424776-9a5e-4b56-8f19-c8135bfc42d8-kube-api-access-jjdvf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405182 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.404992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405182 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.405021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf424776-9a5e-4b56-8f19-c8135bfc42d8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405182 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.405149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405374 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.405320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405374 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.405355    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.405486 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.405467    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.407334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.407315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf424776-9a5e-4b56-8f19-c8135bfc42d8-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.407529 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.407513    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf424776-9a5e-4b56-8f19-c8135bfc42d8-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.413384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.413356    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdvf\" (UniqueName: \"kubernetes.io/projected/cf424776-9a5e-4b56-8f19-c8135bfc42d8-kube-api-access-jjdvf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7\" (UID: \"cf424776-9a5e-4b56-8f19-c8135bfc42d8\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.494734 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.494648    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:37.773415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:37.773390    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"]
Sep 15 11:35:37.775351 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:35:37.775321    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf424776_9a5e_4b56_8f19_c8135bfc42d8.slice/crio-088194d5f9af7fc662ba7f0c066924675f3b0f24303698816f75193ec9316142 WatchSource:0}: Error finding container 088194d5f9af7fc662ba7f0c066924675f3b0f24303698816f75193ec9316142: Status 404 returned error can't find the container with id 088194d5f9af7fc662ba7f0c066924675f3b0f24303698816f75193ec9316142
Sep 15 11:35:38.535597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:38.535558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7" event={"ID":"cf424776-9a5e-4b56-8f19-c8135bfc42d8","Type":"ContainerStarted","Data":"088194d5f9af7fc662ba7f0c066924675f3b0f24303698816f75193ec9316142"}
Sep 15 11:35:39.541698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:39.541654    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7" event={"ID":"cf424776-9a5e-4b56-8f19-c8135bfc42d8","Type":"ContainerStarted","Data":"3122ce10e9c9a56f2f264e78ae468351f24a95ba94b026c74c7a5a4814b5d672"}
Sep 15 11:35:39.562877 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:39.562825    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7" podStartSLOduration=1.475148566 podStartE2EDuration="2.562809831s" totalImagesPullingTime="1.087661265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:35:39.561916899 +0000 UTC m=+633.339593156" watchObservedRunningTime="2026-09-15 11:35:39.562809831 +0000 UTC m=+633.340486110"
Sep 15 11:35:47.495111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:47.495064    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:47.497697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:47.495156    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:47.507923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:47.507897    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:47.590471 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:47.590440    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7"
Sep 15 11:35:56.572854 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.572824    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"]
Sep 15 11:35:56.578404 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.578384    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.580636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.580612    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 11:35:56.585209 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.585185    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"]
Sep 15 11:35:56.677108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.677075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.677256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.677207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d671c9-5631-4ccc-b327-34ce9e347094-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.677256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.677240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.677348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.677269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzl2\" (UniqueName: \"kubernetes.io/projected/d6d671c9-5631-4ccc-b327-34ce9e347094-kube-api-access-crzl2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.677406 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.677389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.677459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.677436    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.777998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.777961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d671c9-5631-4ccc-b327-34ce9e347094-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzl2\" (UniqueName: \"kubernetes.io/projected/d6d671c9-5631-4ccc-b327-34ce9e347094-kube-api-access-crzl2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778159    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778802    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778970 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.778970 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.778950    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.780876 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.780854    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6d671c9-5631-4ccc-b327-34ce9e347094-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.780993 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.780855    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6d671c9-5631-4ccc-b327-34ce9e347094-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.786270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.786249    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzl2\" (UniqueName: \"kubernetes.io/projected/d6d671c9-5631-4ccc-b327-34ce9e347094-kube-api-access-crzl2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-p6dtf\" (UID: \"d6d671c9-5631-4ccc-b327-34ce9e347094\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:56.891801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:56.891705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:35:57.168318 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:57.168294    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"]
Sep 15 11:35:57.170232 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:35:57.170202    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d671c9_5631_4ccc_b327_34ce9e347094.slice/crio-ec996bda02679b35be3b857b96c6acc1f21ecfb87535862f37b62ca834c26ef9 WatchSource:0}: Error finding container ec996bda02679b35be3b857b96c6acc1f21ecfb87535862f37b62ca834c26ef9: Status 404 returned error can't find the container with id ec996bda02679b35be3b857b96c6acc1f21ecfb87535862f37b62ca834c26ef9
Sep 15 11:35:57.628080 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:57.627988    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf" event={"ID":"d6d671c9-5631-4ccc-b327-34ce9e347094","Type":"ContainerStarted","Data":"dd5e7a7ce8229716fdf33e391d623d2985c2df08ae29e9009a43e8a3e4c39e70"}
Sep 15 11:35:57.628080 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:57.628043    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf" event={"ID":"d6d671c9-5631-4ccc-b327-34ce9e347094","Type":"ContainerStarted","Data":"ec996bda02679b35be3b857b96c6acc1f21ecfb87535862f37b62ca834c26ef9"}
Sep 15 11:35:57.648443 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:35:57.648395    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf" podStartSLOduration=1.456932707 podStartE2EDuration="1.648377625s" totalImagesPullingTime="191.444918ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:35:57.647246926 +0000 UTC m=+651.424923187" watchObservedRunningTime="2026-09-15 11:35:57.648377625 +0000 UTC m=+651.426053883"
Sep 15 11:36:06.892813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:06.892778    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:36:06.893251 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:06.892863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:36:06.905662 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:06.905638    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:36:07.687618 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:07.687582    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-p6dtf"
Sep 15 11:36:20.378673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.378634    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"]
Sep 15 11:36:20.381072 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.381053    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.383424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.383405    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 11:36:20.392307 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.392283    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"]
Sep 15 11:36:20.491917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.491867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.492086 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.491944    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.492086 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.492001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.492086 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.492041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsj4g\" (UniqueName: \"kubernetes.io/projected/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-kube-api-access-wsj4g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.492086 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.492071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.492224 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.492133    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.593628 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.593591    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.593740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.593637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.593740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.593658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsj4g\" (UniqueName: \"kubernetes.io/projected/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-kube-api-access-wsj4g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.593740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.593691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.593740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.593727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.593868 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.593783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.594043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.594017    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.594118 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.594026    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.594170 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.594110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.595984 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.595956    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.596216 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.596199    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.601647 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.601629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsj4g\" (UniqueName: \"kubernetes.io/projected/26567ed4-0ae6-46f4-9f02-1aa15d37f41b-kube-api-access-wsj4g\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54\" (UID: \"26567ed4-0ae6-46f4-9f02-1aa15d37f41b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.691931 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.691891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:20.962115 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.962097    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"]
Sep 15 11:36:20.964391 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:36:20.964358    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26567ed4_0ae6_46f4_9f02_1aa15d37f41b.slice/crio-5c4cb44755164700f3e54414c4760ce781cb6b2466ecd0fd3585156c8f354401 WatchSource:0}: Error finding container 5c4cb44755164700f3e54414c4760ce781cb6b2466ecd0fd3585156c8f354401: Status 404 returned error can't find the container with id 5c4cb44755164700f3e54414c4760ce781cb6b2466ecd0fd3585156c8f354401
Sep 15 11:36:20.965703 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:20.965684    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:36:21.739988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:21.739947    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54" event={"ID":"26567ed4-0ae6-46f4-9f02-1aa15d37f41b","Type":"ContainerStarted","Data":"c14141c8cb5e2e36b0611b87949238caaa7c691377aeff2b9f187e4a2651127d"}
Sep 15 11:36:21.740375 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:21.740008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54" event={"ID":"26567ed4-0ae6-46f4-9f02-1aa15d37f41b","Type":"ContainerStarted","Data":"5c4cb44755164700f3e54414c4760ce781cb6b2466ecd0fd3585156c8f354401"}
Sep 15 11:36:21.759386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:21.759335    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54" podStartSLOduration=1.546188056 podStartE2EDuration="1.759318511s" totalImagesPullingTime="213.130455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:36:21.758417465 +0000 UTC m=+675.536093723" watchObservedRunningTime="2026-09-15 11:36:21.759318511 +0000 UTC m=+675.536994781"
Sep 15 11:36:30.692890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:30.692838    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:30.693335 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:30.692913    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:30.709400 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:30.709371    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:30.798575 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:30.798542    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54"
Sep 15 11:36:41.572994 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.572964    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"]
Sep 15 11:36:41.575359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.575340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.577991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.577966    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 11:36:41.589867 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.589843    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"]
Sep 15 11:36:41.683383 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.683346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86td\" (UniqueName: \"kubernetes.io/projected/ccf95788-06f9-452c-8236-e1b0bceae601-kube-api-access-b86td\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.683615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.683530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.683615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.683572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.683749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.683619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf95788-06f9-452c-8236-e1b0bceae601-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.683749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.683689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.683749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.683731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.784976 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.784937    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86td\" (UniqueName: \"kubernetes.io/projected/ccf95788-06f9-452c-8236-e1b0bceae601-kube-api-access-b86td\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf95788-06f9-452c-8236-e1b0bceae601-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785471 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785493    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.785594 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.785528    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.787379 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.787355    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf95788-06f9-452c-8236-e1b0bceae601-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.787653 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.787638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf95788-06f9-452c-8236-e1b0bceae601-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.793200 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.793178    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86td\" (UniqueName: \"kubernetes.io/projected/ccf95788-06f9-452c-8236-e1b0bceae601-kube-api-access-b86td\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc\" (UID: \"ccf95788-06f9-452c-8236-e1b0bceae601\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:41.885390 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:41.885296    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:42.168048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:42.168020    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"]
Sep 15 11:36:42.169970 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:36:42.169933    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccf95788_06f9_452c_8236_e1b0bceae601.slice/crio-968a4d3756f7756e0bfd3f0de79daf5332643220972ec9e991b952a6b34f23c3 WatchSource:0}: Error finding container 968a4d3756f7756e0bfd3f0de79daf5332643220972ec9e991b952a6b34f23c3: Status 404 returned error can't find the container with id 968a4d3756f7756e0bfd3f0de79daf5332643220972ec9e991b952a6b34f23c3
Sep 15 11:36:42.845210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:42.845178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc" event={"ID":"ccf95788-06f9-452c-8236-e1b0bceae601","Type":"ContainerStarted","Data":"561a75912383be46115e41107038cc77538ab3eb5b3d2c32c97bf9660ead263e"}
Sep 15 11:36:42.845210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:42.845215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc" event={"ID":"ccf95788-06f9-452c-8236-e1b0bceae601","Type":"ContainerStarted","Data":"968a4d3756f7756e0bfd3f0de79daf5332643220972ec9e991b952a6b34f23c3"}
Sep 15 11:36:42.863333 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:42.863276    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc" podStartSLOduration=1.700943514 podStartE2EDuration="1.863256816s" totalImagesPullingTime="162.313302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:36:42.862087401 +0000 UTC m=+696.639763658" watchObservedRunningTime="2026-09-15 11:36:42.863256816 +0000 UTC m=+696.640933075"
Sep 15 11:36:51.885835 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:51.885793    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:51.886352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:51.885896    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:51.898942 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:51.898916    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:36:52.909624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:36:52.909596    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc"
Sep 15 11:38:16.116211 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.116173    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64f56c46b4-wz7xb"]
Sep 15 11:38:16.118678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.118662    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.139250 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.139227    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64f56c46b4-wz7xb"]
Sep 15 11:38:16.246506 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.246464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5k6\" (UniqueName: \"kubernetes.io/projected/cc7b13d7-c6a7-43f5-81be-17be7baa5716-kube-api-access-mf5k6\") pod \"authorino-64f56c46b4-wz7xb\" (UID: \"cc7b13d7-c6a7-43f5-81be-17be7baa5716\") " pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.246698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.246657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc7b13d7-c6a7-43f5-81be-17be7baa5716-tls-cert\") pod \"authorino-64f56c46b4-wz7xb\" (UID: \"cc7b13d7-c6a7-43f5-81be-17be7baa5716\") " pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.347707 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.347665    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5k6\" (UniqueName: \"kubernetes.io/projected/cc7b13d7-c6a7-43f5-81be-17be7baa5716-kube-api-access-mf5k6\") pod \"authorino-64f56c46b4-wz7xb\" (UID: \"cc7b13d7-c6a7-43f5-81be-17be7baa5716\") " pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.347893 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.347773    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc7b13d7-c6a7-43f5-81be-17be7baa5716-tls-cert\") pod \"authorino-64f56c46b4-wz7xb\" (UID: \"cc7b13d7-c6a7-43f5-81be-17be7baa5716\") " pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.350207 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.350183    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cc7b13d7-c6a7-43f5-81be-17be7baa5716-tls-cert\") pod \"authorino-64f56c46b4-wz7xb\" (UID: \"cc7b13d7-c6a7-43f5-81be-17be7baa5716\") " pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.359676 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.359655    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5k6\" (UniqueName: \"kubernetes.io/projected/cc7b13d7-c6a7-43f5-81be-17be7baa5716-kube-api-access-mf5k6\") pod \"authorino-64f56c46b4-wz7xb\" (UID: \"cc7b13d7-c6a7-43f5-81be-17be7baa5716\") " pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.427384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.427352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64f56c46b4-wz7xb"
Sep 15 11:38:16.712979 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:16.712949    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64f56c46b4-wz7xb"]
Sep 15 11:38:16.713784 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:16.713762    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc7b13d7_c6a7_43f5_81be_17be7baa5716.slice/crio-4a2f7bb37b04c812335e5529926e4a7ece039fd515fa0ebd69aaf5b7aa8b4f72 WatchSource:0}: Error finding container 4a2f7bb37b04c812335e5529926e4a7ece039fd515fa0ebd69aaf5b7aa8b4f72: Status 404 returned error can't find the container with id 4a2f7bb37b04c812335e5529926e4a7ece039fd515fa0ebd69aaf5b7aa8b4f72
Sep 15 11:38:17.303285 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.303251    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64f56c46b4-wz7xb" event={"ID":"cc7b13d7-c6a7-43f5-81be-17be7baa5716","Type":"ContainerStarted","Data":"94ea768e2c8031ada84657f269524029ce27a3f1b0b635e3d611e0295df64f7a"}
Sep 15 11:38:17.303635 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.303296    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64f56c46b4-wz7xb" event={"ID":"cc7b13d7-c6a7-43f5-81be-17be7baa5716","Type":"ContainerStarted","Data":"4a2f7bb37b04c812335e5529926e4a7ece039fd515fa0ebd69aaf5b7aa8b4f72"}
Sep 15 11:38:17.361639 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.361586    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-64f56c46b4-wz7xb" podStartSLOduration=0.86273036 podStartE2EDuration="1.361568361s" totalImagesPullingTime="498.838001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:17.359187221 +0000 UTC m=+791.136863493" watchObservedRunningTime="2026-09-15 11:38:17.361568361 +0000 UTC m=+791.139244618"
Sep 15 11:38:17.435549 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.435512    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6647f697-hhp77"]
Sep 15 11:38:17.435766 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.435738    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6647f697-hhp77" podUID="77ee4b3e-77e1-47f4-911f-94d79bd7ea74" containerName="authorino" containerID="cri-o://96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6" gracePeriod=30
Sep 15 11:38:17.698340 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.698311    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:38:17.759888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.759798    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mccr\" (UniqueName: \"kubernetes.io/projected/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-kube-api-access-4mccr\") pod \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") "
Sep 15 11:38:17.759888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.759856    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-tls-cert\") pod \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\" (UID: \"77ee4b3e-77e1-47f4-911f-94d79bd7ea74\") "
Sep 15 11:38:17.762637 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.762607    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-kube-api-access-4mccr" (OuterVolumeSpecName: "kube-api-access-4mccr") pod "77ee4b3e-77e1-47f4-911f-94d79bd7ea74" (UID: "77ee4b3e-77e1-47f4-911f-94d79bd7ea74"). InnerVolumeSpecName "kube-api-access-4mccr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:38:17.772306 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.772267    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "77ee4b3e-77e1-47f4-911f-94d79bd7ea74" (UID: "77ee4b3e-77e1-47f4-911f-94d79bd7ea74"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:38:17.861783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.861743    2574 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-tls-cert\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:38:17.861783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:17.861780    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mccr\" (UniqueName: \"kubernetes.io/projected/77ee4b3e-77e1-47f4-911f-94d79bd7ea74-kube-api-access-4mccr\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:38:18.311150 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.311110    2574 generic.go:356] "Generic (PLEG): container finished" podID="77ee4b3e-77e1-47f4-911f-94d79bd7ea74" containerID="96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6" exitCode=0
Sep 15 11:38:18.311626 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.311158    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6647f697-hhp77"
Sep 15 11:38:18.311626 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.311214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6647f697-hhp77" event={"ID":"77ee4b3e-77e1-47f4-911f-94d79bd7ea74","Type":"ContainerDied","Data":"96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6"}
Sep 15 11:38:18.311626 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.311264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6647f697-hhp77" event={"ID":"77ee4b3e-77e1-47f4-911f-94d79bd7ea74","Type":"ContainerDied","Data":"08f3cbf78401ec9e19b3bef37db9b4a047b32bc6a26cf7ab02f1114fe6522002"}
Sep 15 11:38:18.311626 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.311287    2574 scope.go:117] "RemoveContainer" containerID="96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6"
Sep 15 11:38:18.323758 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.323739    2574 scope.go:117] "RemoveContainer" containerID="96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6"
Sep 15 11:38:18.324067 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:38:18.324046    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6\": container with ID starting with 96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6 not found: ID does not exist" containerID="96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6"
Sep 15 11:38:18.324144 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.324074    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6"} err="failed to get container status \"96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6\": rpc error: code = NotFound desc = could not find container \"96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6\": container with ID starting with 96378d1828f6a1dc0c7a772403abb5f6a0aa59cfcb2220c9a003cad90fd926a6 not found: ID does not exist"
Sep 15 11:38:18.365828 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.365785    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6647f697-hhp77"]
Sep 15 11:38:18.378120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.378090    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6647f697-hhp77"]
Sep 15 11:38:18.666296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:18.666259    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77ee4b3e-77e1-47f4-911f-94d79bd7ea74" path="/var/lib/kubelet/pods/77ee4b3e-77e1-47f4-911f-94d79bd7ea74/volumes"
Sep 15 11:38:38.886505 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.886419    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7488-b97ccb"]
Sep 15 11:38:38.886861 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.886802    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77ee4b3e-77e1-47f4-911f-94d79bd7ea74" containerName="authorino"
Sep 15 11:38:38.886861 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.886814    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="77ee4b3e-77e1-47f4-911f-94d79bd7ea74" containerName="authorino"
Sep 15 11:38:38.886932 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.886900    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="77ee4b3e-77e1-47f4-911f-94d79bd7ea74" containerName="authorino"
Sep 15 11:38:38.888695 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.888679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:38.892117 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.892098    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lrtt7\""
Sep 15 11:38:38.954596 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:38.954563    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7488-b97ccb"]
Sep 15 11:38:39.057226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:39.057190    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch6vd\" (UniqueName: \"kubernetes.io/projected/dc83ce67-deea-4273-a866-fa28c5b72568-kube-api-access-ch6vd\") pod \"test-curl-7488-b97ccb\" (UID: \"dc83ce67-deea-4273-a866-fa28c5b72568\") " pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:39.158424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:39.158388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch6vd\" (UniqueName: \"kubernetes.io/projected/dc83ce67-deea-4273-a866-fa28c5b72568-kube-api-access-ch6vd\") pod \"test-curl-7488-b97ccb\" (UID: \"dc83ce67-deea-4273-a866-fa28c5b72568\") " pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:39.167866 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:39.167843    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch6vd\" (UniqueName: \"kubernetes.io/projected/dc83ce67-deea-4273-a866-fa28c5b72568-kube-api-access-ch6vd\") pod \"test-curl-7488-b97ccb\" (UID: \"dc83ce67-deea-4273-a866-fa28c5b72568\") " pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:39.197890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:39.197861    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:39.474420 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:39.474386    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7488-b97ccb"]
Sep 15 11:38:39.474798 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:39.474772    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc83ce67_deea_4273_a866_fa28c5b72568.slice/crio-19ab3392411d7020e562c226e0c013941bcebf9160a7acdb89e6a60f324c623a WatchSource:0}: Error finding container 19ab3392411d7020e562c226e0c013941bcebf9160a7acdb89e6a60f324c623a: Status 404 returned error can't find the container with id 19ab3392411d7020e562c226e0c013941bcebf9160a7acdb89e6a60f324c623a
Sep 15 11:38:40.423082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:40.423046    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7488-b97ccb" event={"ID":"dc83ce67-deea-4273-a866-fa28c5b72568","Type":"ContainerStarted","Data":"19ab3392411d7020e562c226e0c013941bcebf9160a7acdb89e6a60f324c623a"}
Sep 15 11:38:44.441042 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.441007    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"]
Sep 15 11:38:44.444693 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.444674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.450079 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.450057    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-9bd868\""
Sep 15 11:38:44.462968 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.462938    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"]
Sep 15 11:38:44.488988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.488959    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"]
Sep 15 11:38:44.491574 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.491553    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.495848 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.495797    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-9bd868-dockercfg-5dff7\""
Sep 15 11:38:44.508414 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.508384    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"]
Sep 15 11:38:44.508989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.508968    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2f5\" (UniqueName: \"kubernetes.io/projected/2c77cf0d-5c41-44f1-8442-06e5575866a3-kube-api-access-bn2f5\") pod \"payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.509096 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.509081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2afe84-3455-4811-a54b-de889d082ee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.509162 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.509114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j974z\" (UniqueName: \"kubernetes.io/projected/4b2afe84-3455-4811-a54b-de889d082ee1-kube-api-access-j974z\") pod \"payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.509162 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.509138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c77cf0d-5c41-44f1-8442-06e5575866a3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.610191 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.610149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2afe84-3455-4811-a54b-de889d082ee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.610366 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.610211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j974z\" (UniqueName: \"kubernetes.io/projected/4b2afe84-3455-4811-a54b-de889d082ee1-kube-api-access-j974z\") pod \"payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.610366 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.610247    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c77cf0d-5c41-44f1-8442-06e5575866a3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.610366 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.610282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2f5\" (UniqueName: \"kubernetes.io/projected/2c77cf0d-5c41-44f1-8442-06e5575866a3-kube-api-access-bn2f5\") pod \"payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.611170 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.611137    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c77cf0d-5c41-44f1-8442-06e5575866a3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.611308 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.611220    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2afe84-3455-4811-a54b-de889d082ee1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.623613 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.623585    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j974z\" (UniqueName: \"kubernetes.io/projected/4b2afe84-3455-4811-a54b-de889d082ee1-kube-api-access-j974z\") pod \"payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.624136 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.624111    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2f5\" (UniqueName: \"kubernetes.io/projected/2c77cf0d-5c41-44f1-8442-06e5575866a3-kube-api-access-bn2f5\") pod \"payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:44.755862 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.755776    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:44.802835 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:44.802728    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:45.050696 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.050667    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"]
Sep 15 11:38:45.052717 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:45.052686    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2afe84_3455_4811_a54b_de889d082ee1.slice/crio-d16de5990fd1bb6f051d27fb5fe74609e996f5bec5c45120d6b6ad8db789cc17 WatchSource:0}: Error finding container d16de5990fd1bb6f051d27fb5fe74609e996f5bec5c45120d6b6ad8db789cc17: Status 404 returned error can't find the container with id d16de5990fd1bb6f051d27fb5fe74609e996f5bec5c45120d6b6ad8db789cc17
Sep 15 11:38:45.110442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.110413    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"]
Sep 15 11:38:45.112788 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:45.112759    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c77cf0d_5c41_44f1_8442_06e5575866a3.slice/crio-9cebd3022a83d26bf31bd215b0cd8911ea22c1e11b1f73c32b1069bf6c8d18fa WatchSource:0}: Error finding container 9cebd3022a83d26bf31bd215b0cd8911ea22c1e11b1f73c32b1069bf6c8d18fa: Status 404 returned error can't find the container with id 9cebd3022a83d26bf31bd215b0cd8911ea22c1e11b1f73c32b1069bf6c8d18fa
Sep 15 11:38:45.456257 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.455877    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5" event={"ID":"2c77cf0d-5c41-44f1-8442-06e5575866a3","Type":"ContainerStarted","Data":"14f79880ef31c00ac37ec792921ca0601edb25e6991fd109a9dc6620daf07a7b"}
Sep 15 11:38:45.456257 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.455959    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5" event={"ID":"2c77cf0d-5c41-44f1-8442-06e5575866a3","Type":"ContainerStarted","Data":"9cebd3022a83d26bf31bd215b0cd8911ea22c1e11b1f73c32b1069bf6c8d18fa"}
Sep 15 11:38:45.457405 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.457336    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:45.459869 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.459696    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt" event={"ID":"4b2afe84-3455-4811-a54b-de889d082ee1","Type":"ContainerStarted","Data":"d75a29d11e549b3e06786d8467aef6149b3b3d523354604e6fa9eff4b92d566a"}
Sep 15 11:38:45.459869 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.459749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt" event={"ID":"4b2afe84-3455-4811-a54b-de889d082ee1","Type":"ContainerStarted","Data":"d16de5990fd1bb6f051d27fb5fe74609e996f5bec5c45120d6b6ad8db789cc17"}
Sep 15 11:38:45.459869 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.459863    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:45.486994 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.486940    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5" podStartSLOduration=1.48692148 podStartE2EDuration="1.48692148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:45.486332254 +0000 UTC m=+819.264008512" watchObservedRunningTime="2026-09-15 11:38:45.48692148 +0000 UTC m=+819.264597736"
Sep 15 11:38:45.534991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.534936    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt" podStartSLOduration=1.534913808 podStartE2EDuration="1.534913808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:45.531893943 +0000 UTC m=+819.309570201" watchObservedRunningTime="2026-09-15 11:38:45.534913808 +0000 UTC m=+819.312590065"
Sep 15 11:38:45.611919 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.611879    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"]
Sep 15 11:38:45.618296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.618255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.618787 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.618759    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.618969 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.618949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78j67\" (UniqueName: \"kubernetes.io/projected/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-kube-api-access-78j67\") pod \"payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.621026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.620923    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0eb0ef\""
Sep 15 11:38:45.621683 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.621660    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0eb0ef-dockercfg-rrcjr\""
Sep 15 11:38:45.633224 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.633193    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"]
Sep 15 11:38:45.720275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.720152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78j67\" (UniqueName: \"kubernetes.io/projected/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-kube-api-access-78j67\") pod \"payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.720448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.720282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.721275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.721223    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.729461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.729430    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78j67\" (UniqueName: \"kubernetes.io/projected/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-kube-api-access-78j67\") pod \"payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:45.772292 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.772261    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"]
Sep 15 11:38:45.777881 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.777856    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.803530 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.803482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"]
Sep 15 11:38:45.821168 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.821134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747f1974-0432-4f5c-ac0d-9ab12a0074cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.821359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.821240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5g4\" (UniqueName: \"kubernetes.io/projected/747f1974-0432-4f5c-ac0d-9ab12a0074cc-kube-api-access-xk5g4\") pod \"payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.922249 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.922211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747f1974-0432-4f5c-ac0d-9ab12a0074cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.922410 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.922264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5g4\" (UniqueName: \"kubernetes.io/projected/747f1974-0432-4f5c-ac0d-9ab12a0074cc-kube-api-access-xk5g4\") pod \"payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.922954 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.922932    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747f1974-0432-4f5c-ac0d-9ab12a0074cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.933159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.933108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5g4\" (UniqueName: \"kubernetes.io/projected/747f1974-0432-4f5c-ac0d-9ab12a0074cc-kube-api-access-xk5g4\") pod \"payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:45.952772 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:45.952747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:46.091177 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.091144    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:46.269279 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.269181    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"]
Sep 15 11:38:46.272032 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:46.272004    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3c4aea_a301_496c_8eae_dc59ed5dfe1b.slice/crio-65dd160105c06b1e328c157d6bf2fe6c19da214309449ecc5ce215f967e00b21 WatchSource:0}: Error finding container 65dd160105c06b1e328c157d6bf2fe6c19da214309449ecc5ce215f967e00b21: Status 404 returned error can't find the container with id 65dd160105c06b1e328c157d6bf2fe6c19da214309449ecc5ce215f967e00b21
Sep 15 11:38:46.366862 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.366832    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"]
Sep 15 11:38:46.368799 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:46.368772    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747f1974_0432_4f5c_ac0d_9ab12a0074cc.slice/crio-3821621ad5ba69c68d5c10fc0aefa7588c790b7bc05954b0ebdfea70209abaec WatchSource:0}: Error finding container 3821621ad5ba69c68d5c10fc0aefa7588c790b7bc05954b0ebdfea70209abaec: Status 404 returned error can't find the container with id 3821621ad5ba69c68d5c10fc0aefa7588c790b7bc05954b0ebdfea70209abaec
Sep 15 11:38:46.467643 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.467609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5" event={"ID":"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b","Type":"ContainerStarted","Data":"b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2"}
Sep 15 11:38:46.468101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.467683    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:38:46.468101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.467708    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5" event={"ID":"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b","Type":"ContainerStarted","Data":"65dd160105c06b1e328c157d6bf2fe6c19da214309449ecc5ce215f967e00b21"}
Sep 15 11:38:46.469390 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.469368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw" event={"ID":"747f1974-0432-4f5c-ac0d-9ab12a0074cc","Type":"ContainerStarted","Data":"0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895"}
Sep 15 11:38:46.469528 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.469404    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw" event={"ID":"747f1974-0432-4f5c-ac0d-9ab12a0074cc","Type":"ContainerStarted","Data":"3821621ad5ba69c68d5c10fc0aefa7588c790b7bc05954b0ebdfea70209abaec"}
Sep 15 11:38:46.469528 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.469464    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:46.471082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.471059    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7488-b97ccb" event={"ID":"dc83ce67-deea-4273-a866-fa28c5b72568","Type":"ContainerStarted","Data":"8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1"}
Sep 15 11:38:46.486899 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.486855    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5" podStartSLOduration=1.486841233 podStartE2EDuration="1.486841233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:46.483953324 +0000 UTC m=+820.261629574" watchObservedRunningTime="2026-09-15 11:38:46.486841233 +0000 UTC m=+820.264517521"
Sep 15 11:38:46.501571 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.501527    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw" podStartSLOduration=1.50148579 podStartE2EDuration="1.50148579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:46.499345683 +0000 UTC m=+820.277021939" watchObservedRunningTime="2026-09-15 11:38:46.50148579 +0000 UTC m=+820.279162047"
Sep 15 11:38:46.515967 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:46.515926    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7488-b97ccb" podStartSLOduration=1.989572519 podStartE2EDuration="8.515913169s" totalImagesPullingTime="6.52634065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:46.514354369 +0000 UTC m=+820.292030625" watchObservedRunningTime="2026-09-15 11:38:46.515913169 +0000 UTC m=+820.293589403"
Sep 15 11:38:47.097388 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:47.097300    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7488-b97ccb"]
Sep 15 11:38:47.104843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:47.104816    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7488-b97ccb"]
Sep 15 11:38:47.477733 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:38:47.477700    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7488-b97ccb" container="test-curl-7488-b97ccb"
Sep 15 11:38:48.481757 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:48.481708    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7488-b97ccb" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" containerName="test-curl-7488-b97ccb" containerID="cri-o://8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1" gracePeriod=2
Sep 15 11:38:48.483715 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:38:48.483675    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7488-b97ccb\" is forbidden: User \"system:node:ip-10-0-142-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-54.ec2.internal' and this object" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:50.675147 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:50.675122    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:50.768527 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:50.768431    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ch6vd\" (UniqueName: \"kubernetes.io/projected/dc83ce67-deea-4273-a866-fa28c5b72568-kube-api-access-ch6vd\") pod \"dc83ce67-deea-4273-a866-fa28c5b72568\" (UID: \"dc83ce67-deea-4273-a866-fa28c5b72568\") "
Sep 15 11:38:50.770560 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:50.770529    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc83ce67-deea-4273-a866-fa28c5b72568-kube-api-access-ch6vd" (OuterVolumeSpecName: "kube-api-access-ch6vd") pod "dc83ce67-deea-4273-a866-fa28c5b72568" (UID: "dc83ce67-deea-4273-a866-fa28c5b72568"). InnerVolumeSpecName "kube-api-access-ch6vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:38:50.869588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:50.869554    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch6vd\" (UniqueName: \"kubernetes.io/projected/dc83ce67-deea-4273-a866-fa28c5b72568-kube-api-access-ch6vd\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:38:51.498398 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:51.498358    2574 generic.go:356] "Generic (PLEG): container finished" podID="dc83ce67-deea-4273-a866-fa28c5b72568" containerID="8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1" exitCode=137
Sep 15 11:38:51.498588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:51.498408    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:51.498588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:51.498454    2574 scope.go:117] "RemoveContainer" containerID="8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1"
Sep 15 11:38:51.507837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:51.507820    2574 scope.go:117] "RemoveContainer" containerID="8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1"
Sep 15 11:38:51.508105 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:38:51.508088    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1\": container with ID starting with 8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1 not found: ID does not exist" containerID="8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1"
Sep 15 11:38:51.508159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:51.508112    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1"} err="failed to get container status \"8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1\": rpc error: code = NotFound desc = could not find container \"8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1\": container with ID starting with 8ae200e30ebdb752325d3ef7c394e16666c5c843659fc10e5c7e360f19a177f1 not found: ID does not exist"
Sep 15 11:38:51.508915 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:38:51.508888    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7488-b97ccb\" is forbidden: User \"system:node:ip-10-0-142-54.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-54.ec2.internal' and this object" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" pod="opendatahub/test-curl-7488-b97ccb"
Sep 15 11:38:52.665458 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:52.665424    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" path="/var/lib/kubelet/pods/dc83ce67-deea-4273-a866-fa28c5b72568/volumes"
Sep 15 11:38:54.860939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.860901    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"]
Sep 15 11:38:54.861511 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.861480    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" containerName="test-curl-7488-b97ccb"
Sep 15 11:38:54.861568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.861519    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" containerName="test-curl-7488-b97ccb"
Sep 15 11:38:54.861682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.861669    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc83ce67-deea-4273-a866-fa28c5b72568" containerName="test-curl-7488-b97ccb"
Sep 15 11:38:54.866211 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.866196    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:54.868472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.868450    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9f77ab79\""
Sep 15 11:38:54.868873 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.868854    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9f77ab79-dockercfg-w2sth\""
Sep 15 11:38:54.877223 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.877200    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"]
Sep 15 11:38:54.907677 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.907647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630d87af-e7fa-4ead-82d4-09431aa9a581-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") " pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:54.907822 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.907736    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52g4c\" (UniqueName: \"kubernetes.io/projected/630d87af-e7fa-4ead-82d4-09431aa9a581-kube-api-access-52g4c\") pod \"payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") " pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:54.912239 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.912206    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"]
Sep 15 11:38:54.915847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.915829    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:54.927237 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:54.927216    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"]
Sep 15 11:38:55.009106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.009078    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630d87af-e7fa-4ead-82d4-09431aa9a581-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") " pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:55.009261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.009126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52g4c\" (UniqueName: \"kubernetes.io/projected/630d87af-e7fa-4ead-82d4-09431aa9a581-kube-api-access-52g4c\") pod \"payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") " pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:55.009261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.009158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4fx\" (UniqueName: \"kubernetes.io/projected/f6b65949-a213-453a-8dfa-06f46457559b-kube-api-access-pc4fx\") pod \"payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.009261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.009186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b65949-a213-453a-8dfa-06f46457559b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.009897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.009876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630d87af-e7fa-4ead-82d4-09431aa9a581-plugins-config-volume\") pod \"payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") " pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:55.017204 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.017180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52g4c\" (UniqueName: \"kubernetes.io/projected/630d87af-e7fa-4ead-82d4-09431aa9a581-kube-api-access-52g4c\") pod \"payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") " pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:55.110190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.110155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4fx\" (UniqueName: \"kubernetes.io/projected/f6b65949-a213-453a-8dfa-06f46457559b-kube-api-access-pc4fx\") pod \"payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.110190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.110198    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b65949-a213-453a-8dfa-06f46457559b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.110891 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.110873    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b65949-a213-453a-8dfa-06f46457559b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.118883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.118826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4fx\" (UniqueName: \"kubernetes.io/projected/f6b65949-a213-453a-8dfa-06f46457559b-kube-api-access-pc4fx\") pod \"payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.177483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.177444    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:55.226820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.226193    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:55.464184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.464134    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"]
Sep 15 11:38:55.467017 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:55.466988    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630d87af_e7fa_4ead_82d4_09431aa9a581.slice/crio-f31dada6a793bf305f54ae78773e5ce0ab237f85a6b0b035fdd7a18c28aaf6b1 WatchSource:0}: Error finding container f31dada6a793bf305f54ae78773e5ce0ab237f85a6b0b035fdd7a18c28aaf6b1: Status 404 returned error can't find the container with id f31dada6a793bf305f54ae78773e5ce0ab237f85a6b0b035fdd7a18c28aaf6b1
Sep 15 11:38:55.524563 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.524528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2" event={"ID":"630d87af-e7fa-4ead-82d4-09431aa9a581","Type":"ContainerStarted","Data":"f31dada6a793bf305f54ae78773e5ce0ab237f85a6b0b035fdd7a18c28aaf6b1"}
Sep 15 11:38:55.528627 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:55.528601    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"]
Sep 15 11:38:55.530531 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:38:55.530487    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b65949_a213_453a_8dfa_06f46457559b.slice/crio-2f111b8597c9cbbc22b9b1d5aafccc28ff560377d7d84eaec8e95260e4c95f01 WatchSource:0}: Error finding container 2f111b8597c9cbbc22b9b1d5aafccc28ff560377d7d84eaec8e95260e4c95f01: Status 404 returned error can't find the container with id 2f111b8597c9cbbc22b9b1d5aafccc28ff560377d7d84eaec8e95260e4c95f01
Sep 15 11:38:56.473386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.473355    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:38:56.473830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.473571    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:38:56.532845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.532802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl" event={"ID":"f6b65949-a213-453a-8dfa-06f46457559b","Type":"ContainerStarted","Data":"1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16"}
Sep 15 11:38:56.533043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.532873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl" event={"ID":"f6b65949-a213-453a-8dfa-06f46457559b","Type":"ContainerStarted","Data":"2f111b8597c9cbbc22b9b1d5aafccc28ff560377d7d84eaec8e95260e4c95f01"}
Sep 15 11:38:56.533043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.532906    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:38:56.535036 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.535010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2" event={"ID":"630d87af-e7fa-4ead-82d4-09431aa9a581","Type":"ContainerStarted","Data":"28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0"}
Sep 15 11:38:56.535254 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.535227    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:38:56.565082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.565016    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl" podStartSLOduration=2.564993595 podStartE2EDuration="2.564993595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:56.563875279 +0000 UTC m=+830.341551536" watchObservedRunningTime="2026-09-15 11:38:56.564993595 +0000 UTC m=+830.342669853"
Sep 15 11:38:56.587581 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:56.587528    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2" podStartSLOduration=2.587481624 podStartE2EDuration="2.587481624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:38:56.585672008 +0000 UTC m=+830.363348265" watchObservedRunningTime="2026-09-15 11:38:56.587481624 +0000 UTC m=+830.365157881"
Sep 15 11:38:57.479375 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:57.479347    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:38:57.479777 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:38:57.479414    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:39:00.452074 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.452035    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"]
Sep 15 11:39:00.456243 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.456219    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.461452 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.461430    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-e114ba\""
Sep 15 11:39:00.461565 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.461554    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-e114ba-dockercfg-jv4bm\""
Sep 15 11:39:00.468229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.468206    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"]
Sep 15 11:39:00.563413 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.563373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e0b33-563f-4827-bfc2-14ddbdc53438-plugins-config-volume\") pod \"payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") " pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.563568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.563415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8c8n\" (UniqueName: \"kubernetes.io/projected/ab8e0b33-563f-4827-bfc2-14ddbdc53438-kube-api-access-f8c8n\") pod \"payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") " pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.664531 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.664489    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e0b33-563f-4827-bfc2-14ddbdc53438-plugins-config-volume\") pod \"payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") " pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.664531 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.664532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8c8n\" (UniqueName: \"kubernetes.io/projected/ab8e0b33-563f-4827-bfc2-14ddbdc53438-kube-api-access-f8c8n\") pod \"payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") " pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.665144 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.665123    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e0b33-563f-4827-bfc2-14ddbdc53438-plugins-config-volume\") pod \"payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") " pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.677458 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.677434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8c8n\" (UniqueName: \"kubernetes.io/projected/ab8e0b33-563f-4827-bfc2-14ddbdc53438-kube-api-access-f8c8n\") pod \"payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") " pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:00.768317 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:00.768245    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:01.052575 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:01.052549    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"]
Sep 15 11:39:01.054352 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:39:01.054325    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab8e0b33_563f_4827_bfc2_14ddbdc53438.slice/crio-f1bde4c24643381d7c440bca63f8315063353714b44ffce6299bd10cc9c5bbe7 WatchSource:0}: Error finding container f1bde4c24643381d7c440bca63f8315063353714b44ffce6299bd10cc9c5bbe7: Status 404 returned error can't find the container with id f1bde4c24643381d7c440bca63f8315063353714b44ffce6299bd10cc9c5bbe7
Sep 15 11:39:01.564218 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:01.564181    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27" event={"ID":"ab8e0b33-563f-4827-bfc2-14ddbdc53438","Type":"ContainerStarted","Data":"e2977e5c6d22f0457c4f8282e9d42f7e3a74a21251b26462a04d7bdfbb725ce7"}
Sep 15 11:39:01.564697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:01.564239    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27" event={"ID":"ab8e0b33-563f-4827-bfc2-14ddbdc53438","Type":"ContainerStarted","Data":"f1bde4c24643381d7c440bca63f8315063353714b44ffce6299bd10cc9c5bbe7"}
Sep 15 11:39:01.564697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:01.564288    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:01.585754 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:01.585694    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27" podStartSLOduration=1.5856783700000001 podStartE2EDuration="1.58567837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:01.584205986 +0000 UTC m=+835.361882243" watchObservedRunningTime="2026-09-15 11:39:01.58567837 +0000 UTC m=+835.363354627"
Sep 15 11:39:02.765966 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.765929    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"]
Sep 15 11:39:02.769719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.769697    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:02.772029 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.772009    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1358dc\""
Sep 15 11:39:02.772151 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.772123    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1358dc-dockercfg-5n8vg\""
Sep 15 11:39:02.781456 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.781427    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"]
Sep 15 11:39:02.884054 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.884020    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"]
Sep 15 11:39:02.887200 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.887178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aee0ab-904e-4ff0-8f17-a311d714b22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:02.887320 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.887221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jft4h\" (UniqueName: \"kubernetes.io/projected/58aee0ab-904e-4ff0-8f17-a311d714b22f-kube-api-access-jft4h\") pod \"payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:02.887520 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.887491    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:02.902271 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.902245    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"]
Sep 15 11:39:02.988137 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.988095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aee0ab-904e-4ff0-8f17-a311d714b22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:02.988332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.988157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jft4h\" (UniqueName: \"kubernetes.io/projected/58aee0ab-904e-4ff0-8f17-a311d714b22f-kube-api-access-jft4h\") pod \"payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:02.988332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.988204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626d9add-f766-4b85-bea5-8455f4b2882e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:02.988332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.988287    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brt7\" (UniqueName: \"kubernetes.io/projected/626d9add-f766-4b85-bea5-8455f4b2882e-kube-api-access-7brt7\") pod \"payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:02.988872 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:02.988852    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aee0ab-904e-4ff0-8f17-a311d714b22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:03.006171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.006145    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jft4h\" (UniqueName: \"kubernetes.io/projected/58aee0ab-904e-4ff0-8f17-a311d714b22f-kube-api-access-jft4h\") pod \"payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:03.081082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.081002    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:03.089719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.089614    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brt7\" (UniqueName: \"kubernetes.io/projected/626d9add-f766-4b85-bea5-8455f4b2882e-kube-api-access-7brt7\") pod \"payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:03.090060 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.090034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626d9add-f766-4b85-bea5-8455f4b2882e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:03.091144 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.091122    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626d9add-f766-4b85-bea5-8455f4b2882e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:03.097970 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.097946    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brt7\" (UniqueName: \"kubernetes.io/projected/626d9add-f766-4b85-bea5-8455f4b2882e-kube-api-access-7brt7\") pod \"payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:03.197256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.197209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:03.352675 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.352648    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"]
Sep 15 11:39:03.354453 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:39:03.354414    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aee0ab_904e_4ff0_8f17_a311d714b22f.slice/crio-f8fafd24f978e83c1e57d2ca76f32b216fc1b0cf6dc0d60e5863ad5d9222f001 WatchSource:0}: Error finding container f8fafd24f978e83c1e57d2ca76f32b216fc1b0cf6dc0d60e5863ad5d9222f001: Status 404 returned error can't find the container with id f8fafd24f978e83c1e57d2ca76f32b216fc1b0cf6dc0d60e5863ad5d9222f001
Sep 15 11:39:03.472775 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.472750    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"]
Sep 15 11:39:03.474374 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:39:03.474343    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod626d9add_f766_4b85_bea5_8455f4b2882e.slice/crio-75ae9af985daff1d7496eb72f5d326c4de973c7c890474097084aa83b045fc79 WatchSource:0}: Error finding container 75ae9af985daff1d7496eb72f5d326c4de973c7c890474097084aa83b045fc79: Status 404 returned error can't find the container with id 75ae9af985daff1d7496eb72f5d326c4de973c7c890474097084aa83b045fc79
Sep 15 11:39:03.575833 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.575797    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh" event={"ID":"58aee0ab-904e-4ff0-8f17-a311d714b22f","Type":"ContainerStarted","Data":"a7ef2c7856f318c9e3bdad2e3174ce29b6980fa2b2ae54924d53a880f673bf09"}
Sep 15 11:39:03.575979 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.575853    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh" event={"ID":"58aee0ab-904e-4ff0-8f17-a311d714b22f","Type":"ContainerStarted","Data":"f8fafd24f978e83c1e57d2ca76f32b216fc1b0cf6dc0d60e5863ad5d9222f001"}
Sep 15 11:39:03.576267 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.576245    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:03.577803 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.577781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt" event={"ID":"626d9add-f766-4b85-bea5-8455f4b2882e","Type":"ContainerStarted","Data":"c60a7f9caa1f06446d3991f95c7d2064c8cba49294041475f642ec14ee47df84"}
Sep 15 11:39:03.577883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.577816    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt" event={"ID":"626d9add-f766-4b85-bea5-8455f4b2882e","Type":"ContainerStarted","Data":"75ae9af985daff1d7496eb72f5d326c4de973c7c890474097084aa83b045fc79"}
Sep 15 11:39:03.577883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.577839    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:03.593926 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.593876    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh" podStartSLOduration=1.593858788 podStartE2EDuration="1.593858788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:03.591320758 +0000 UTC m=+837.368997014" watchObservedRunningTime="2026-09-15 11:39:03.593858788 +0000 UTC m=+837.371535044"
Sep 15 11:39:03.606296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:03.606200    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt" podStartSLOduration=1.606178882 podStartE2EDuration="1.606178882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:03.604918499 +0000 UTC m=+837.382594755" watchObservedRunningTime="2026-09-15 11:39:03.606178882 +0000 UTC m=+837.383855139"
Sep 15 11:39:07.544407 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:07.544367    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:39:07.544827 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:07.544460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:39:12.571973 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:12.571936    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:39:14.585691 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:14.585658    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:39:14.586061 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:14.585727    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:39:24.555145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.555108    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"]
Sep 15 11:39:24.562792 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.562763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.566125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.565843    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9bd868\"/\"openshift-service-ca.crt\""
Sep 15 11:39:24.566125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.565870    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9bd868\"/\"kube-root-ca.crt\""
Sep 15 11:39:24.566125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.565843    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bd868\"/\"default-dockercfg-d7rvg\""
Sep 15 11:39:24.566125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.566033    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bd868\"/\"simulator-w0-9bd868-kserve-self-signed-certs\""
Sep 15 11:39:24.568742 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.568716    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"]
Sep 15 11:39:24.705710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.705671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tls-certs\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.705710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.705713    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznnf\" (UniqueName: \"kubernetes.io/projected/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-kube-api-access-hznnf\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.705951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.705739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-home\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.705951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.705839    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-dshm\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.705951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.705872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tmp-dir\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.705951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.705921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-model-cache\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807113 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807018    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-home\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807292 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-dshm\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807292 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tmp-dir\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807292 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807217    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-model-cache\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807570 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tls-certs\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807544    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-home\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807588    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznnf\" (UniqueName: \"kubernetes.io/projected/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-kube-api-access-hznnf\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807601    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tmp-dir\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.807648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.807626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-model-cache\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.809663 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.809639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-dshm\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.809923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.809908    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tls-certs\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.815698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.815679    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznnf\" (UniqueName: \"kubernetes.io/projected/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-kube-api-access-hznnf\") pod \"simulator-w0-9bd868-kserve-757ff85c9d-jwpm9\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") " pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:24.873684 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:24.873645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:25.163597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:25.163570    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"]
Sep 15 11:39:25.163735 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:39:25.163714    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b5d9369_d996_4df8_b073_5ae2a20cc3e4.slice/crio-565d58c5fd3c350dde9b524ac8a0bcac85dd35151d380a0b1dbab61ae9b93e1a WatchSource:0}: Error finding container 565d58c5fd3c350dde9b524ac8a0bcac85dd35151d380a0b1dbab61ae9b93e1a: Status 404 returned error can't find the container with id 565d58c5fd3c350dde9b524ac8a0bcac85dd35151d380a0b1dbab61ae9b93e1a
Sep 15 11:39:25.697281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:25.697248    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9" event={"ID":"2b5d9369-d996-4df8-b073-5ae2a20cc3e4","Type":"ContainerStarted","Data":"363df011078b16b2ca58afb361f5bc6727f11d92dfb15dcb8a0433f480763e9f"}
Sep 15 11:39:25.697730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:25.697291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9" event={"ID":"2b5d9369-d996-4df8-b073-5ae2a20cc3e4","Type":"ContainerStarted","Data":"565d58c5fd3c350dde9b524ac8a0bcac85dd35151d380a0b1dbab61ae9b93e1a"}
Sep 15 11:39:25.716958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:25.716896    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9" podStartSLOduration=1.716879181 podStartE2EDuration="1.716879181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:25.714080101 +0000 UTC m=+859.491756358" watchObservedRunningTime="2026-09-15 11:39:25.716879181 +0000 UTC m=+859.494555439"
Sep 15 11:39:32.301044 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.301000    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"]
Sep 15 11:39:32.301489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.301286    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2" podUID="630d87af-e7fa-4ead-82d4-09431aa9a581" containerName="payload-processing" containerID="cri-o://28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0" gracePeriod=30
Sep 15 11:39:32.313858 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.313808    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"]
Sep 15 11:39:32.314066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.314041    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl" podUID="f6b65949-a213-453a-8dfa-06f46457559b" containerName="payload-pre-processing" containerID="cri-o://1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16" gracePeriod=30
Sep 15 11:39:32.621055 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.620986    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:39:32.625314 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.625286    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:39:32.690695 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.690658    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b65949-a213-453a-8dfa-06f46457559b-plugins-config-volume\") pod \"f6b65949-a213-453a-8dfa-06f46457559b\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") "
Sep 15 11:39:32.690890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.690734    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630d87af-e7fa-4ead-82d4-09431aa9a581-plugins-config-volume\") pod \"630d87af-e7fa-4ead-82d4-09431aa9a581\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") "
Sep 15 11:39:32.690890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.690785    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc4fx\" (UniqueName: \"kubernetes.io/projected/f6b65949-a213-453a-8dfa-06f46457559b-kube-api-access-pc4fx\") pod \"f6b65949-a213-453a-8dfa-06f46457559b\" (UID: \"f6b65949-a213-453a-8dfa-06f46457559b\") "
Sep 15 11:39:32.690890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.690884    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52g4c\" (UniqueName: \"kubernetes.io/projected/630d87af-e7fa-4ead-82d4-09431aa9a581-kube-api-access-52g4c\") pod \"630d87af-e7fa-4ead-82d4-09431aa9a581\" (UID: \"630d87af-e7fa-4ead-82d4-09431aa9a581\") "
Sep 15 11:39:32.691111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.691084    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b65949-a213-453a-8dfa-06f46457559b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6b65949-a213-453a-8dfa-06f46457559b" (UID: "f6b65949-a213-453a-8dfa-06f46457559b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:39:32.691197 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.691145    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630d87af-e7fa-4ead-82d4-09431aa9a581-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "630d87af-e7fa-4ead-82d4-09431aa9a581" (UID: "630d87af-e7fa-4ead-82d4-09431aa9a581"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:39:32.691348 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.691327    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b65949-a213-453a-8dfa-06f46457559b-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:39:32.691529 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.691510    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/630d87af-e7fa-4ead-82d4-09431aa9a581-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:39:32.693313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.693293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b65949-a213-453a-8dfa-06f46457559b-kube-api-access-pc4fx" (OuterVolumeSpecName: "kube-api-access-pc4fx") pod "f6b65949-a213-453a-8dfa-06f46457559b" (UID: "f6b65949-a213-453a-8dfa-06f46457559b"). InnerVolumeSpecName "kube-api-access-pc4fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:39:32.693398 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.693318    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630d87af-e7fa-4ead-82d4-09431aa9a581-kube-api-access-52g4c" (OuterVolumeSpecName: "kube-api-access-52g4c") pod "630d87af-e7fa-4ead-82d4-09431aa9a581" (UID: "630d87af-e7fa-4ead-82d4-09431aa9a581"). InnerVolumeSpecName "kube-api-access-52g4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:39:32.737655 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.737620    2574 generic.go:356] "Generic (PLEG): container finished" podID="f6b65949-a213-453a-8dfa-06f46457559b" containerID="1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16" exitCode=0
Sep 15 11:39:32.737820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.737671    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"
Sep 15 11:39:32.737820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.737706    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl" event={"ID":"f6b65949-a213-453a-8dfa-06f46457559b","Type":"ContainerDied","Data":"1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16"}
Sep 15 11:39:32.737820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.737759    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl" event={"ID":"f6b65949-a213-453a-8dfa-06f46457559b","Type":"ContainerDied","Data":"2f111b8597c9cbbc22b9b1d5aafccc28ff560377d7d84eaec8e95260e4c95f01"}
Sep 15 11:39:32.737820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.737787    2574 scope.go:117] "RemoveContainer" containerID="1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16"
Sep 15 11:39:32.739293 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.739262    2574 generic.go:356] "Generic (PLEG): container finished" podID="630d87af-e7fa-4ead-82d4-09431aa9a581" containerID="28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0" exitCode=0
Sep 15 11:39:32.739412 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.739298    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"
Sep 15 11:39:32.739412 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.739329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2" event={"ID":"630d87af-e7fa-4ead-82d4-09431aa9a581","Type":"ContainerDied","Data":"28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0"}
Sep 15 11:39:32.739412 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.739360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2" event={"ID":"630d87af-e7fa-4ead-82d4-09431aa9a581","Type":"ContainerDied","Data":"f31dada6a793bf305f54ae78773e5ce0ab237f85a6b0b035fdd7a18c28aaf6b1"}
Sep 15 11:39:32.747973 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.747949    2574 scope.go:117] "RemoveContainer" containerID="1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16"
Sep 15 11:39:32.748224 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:39:32.748201    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16\": container with ID starting with 1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16 not found: ID does not exist" containerID="1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16"
Sep 15 11:39:32.748310 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.748230    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16"} err="failed to get container status \"1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16\": rpc error: code = NotFound desc = could not find container \"1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16\": container with ID starting with 1f8ae5f6c1d9b4544717fb41dbb0b00a116994fda77e144628cf9c8a7b9afe16 not found: ID does not exist"
Sep 15 11:39:32.748310 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.748246    2574 scope.go:117] "RemoveContainer" containerID="28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0"
Sep 15 11:39:32.756581 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.756556    2574 scope.go:117] "RemoveContainer" containerID="28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0"
Sep 15 11:39:32.756886 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:39:32.756864    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0\": container with ID starting with 28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0 not found: ID does not exist" containerID="28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0"
Sep 15 11:39:32.756965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.756894    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0"} err="failed to get container status \"28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0\": rpc error: code = NotFound desc = could not find container \"28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0\": container with ID starting with 28d27edbe1816289732bba5dd453999edc781d534854bea830a3b78f53e74ac0 not found: ID does not exist"
Sep 15 11:39:32.769961 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.769920    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"]
Sep 15 11:39:32.771917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.771889    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9f77ab79-54769cf47d-hnljl"]
Sep 15 11:39:32.784480 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.784453    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"]
Sep 15 11:39:32.789461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.789437    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9f77ab79-684cf5db7-n62j2"]
Sep 15 11:39:32.792675 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.792657    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52g4c\" (UniqueName: \"kubernetes.io/projected/630d87af-e7fa-4ead-82d4-09431aa9a581-kube-api-access-52g4c\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:39:32.792731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:32.792677    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4fx\" (UniqueName: \"kubernetes.io/projected/f6b65949-a213-453a-8dfa-06f46457559b-kube-api-access-pc4fx\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:39:33.063328 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063294    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"]
Sep 15 11:39:33.063755 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063742    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b65949-a213-453a-8dfa-06f46457559b" containerName="payload-pre-processing"
Sep 15 11:39:33.063810 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063758    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b65949-a213-453a-8dfa-06f46457559b" containerName="payload-pre-processing"
Sep 15 11:39:33.063810 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063786    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630d87af-e7fa-4ead-82d4-09431aa9a581" containerName="payload-processing"
Sep 15 11:39:33.063810 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063792    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="630d87af-e7fa-4ead-82d4-09431aa9a581" containerName="payload-processing"
Sep 15 11:39:33.063916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063859    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="630d87af-e7fa-4ead-82d4-09431aa9a581" containerName="payload-processing"
Sep 15 11:39:33.063916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.063869    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b65949-a213-453a-8dfa-06f46457559b" containerName="payload-pre-processing"
Sep 15 11:39:33.066956 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.066935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.071262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.071238    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"default-dockercfg-q727f\""
Sep 15 11:39:33.075832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.075809    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"openshift-service-ca.crt\""
Sep 15 11:39:33.075979 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.075817    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"kube-root-ca.crt\""
Sep 15 11:39:33.076205 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.076191    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"simulator-w4-2fab1f-kserve-self-signed-certs\""
Sep 15 11:39:33.088353 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.088324    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"]
Sep 15 11:39:33.096565 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.096536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tmp-dir\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.096719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.096576    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-home\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.096719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.096642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-model-cache\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.096719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.096697    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqdww\" (UniqueName: \"kubernetes.io/projected/27d8e103-544d-4ba8-b52d-c57abe5c48b4-kube-api-access-mqdww\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.096880 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.096726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-dshm\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.096880 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.096763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tls-certs\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198158 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198112    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tmp-dir\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198158 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198157    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-home\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198218    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-model-cache\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqdww\" (UniqueName: \"kubernetes.io/projected/27d8e103-544d-4ba8-b52d-c57abe5c48b4-kube-api-access-mqdww\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198333    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-dshm\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tls-certs\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tmp-dir\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-model-cache\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.198749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.198732    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-home\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.200451 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.200424    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-dshm\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.200761 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.200745    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tls-certs\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.207346 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.207323    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqdww\" (UniqueName: \"kubernetes.io/projected/27d8e103-544d-4ba8-b52d-c57abe5c48b4-kube-api-access-mqdww\") pod \"simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") " pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.377349 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.377312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:33.670415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.670388    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"]
Sep 15 11:39:33.673068 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:39:33.673039    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27d8e103_544d_4ba8_b52d_c57abe5c48b4.slice/crio-76b98af503176d324359dfe43397ba19ebce59f1038e5d6083182d2e7c852d78 WatchSource:0}: Error finding container 76b98af503176d324359dfe43397ba19ebce59f1038e5d6083182d2e7c852d78: Status 404 returned error can't find the container with id 76b98af503176d324359dfe43397ba19ebce59f1038e5d6083182d2e7c852d78
Sep 15 11:39:33.746481 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.746456    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6" event={"ID":"27d8e103-544d-4ba8-b52d-c57abe5c48b4","Type":"ContainerStarted","Data":"adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d"}
Sep 15 11:39:33.746584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:33.746524    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6" event={"ID":"27d8e103-544d-4ba8-b52d-c57abe5c48b4","Type":"ContainerStarted","Data":"76b98af503176d324359dfe43397ba19ebce59f1038e5d6083182d2e7c852d78"}
Sep 15 11:39:34.665630 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:34.665598    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630d87af-e7fa-4ead-82d4-09431aa9a581" path="/var/lib/kubelet/pods/630d87af-e7fa-4ead-82d4-09431aa9a581/volumes"
Sep 15 11:39:34.666010 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:34.666001    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b65949-a213-453a-8dfa-06f46457559b" path="/var/lib/kubelet/pods/f6b65949-a213-453a-8dfa-06f46457559b/volumes"
Sep 15 11:39:34.790649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:34.790596    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6" podStartSLOduration=1.790578715 podStartE2EDuration="1.790578715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:34.788807202 +0000 UTC m=+868.566483460" watchObservedRunningTime="2026-09-15 11:39:34.790578715 +0000 UTC m=+868.568254971"
Sep 15 11:39:34.874797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:34.874763    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:34.874797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:34.874807    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:34.887705 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:34.887677    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:35.769659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:35.769628    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:39:39.170751 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.170715    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"]
Sep 15 11:39:39.175881 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.175863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.178324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.178300    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fc7a9035-dockercfg-wm9w6\""
Sep 15 11:39:39.178730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.178712    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fc7a9035\""
Sep 15 11:39:39.188392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.188365    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"]
Sep 15 11:39:39.258973 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.258930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d2feb0-2647-4011-a902-0135762b43a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.259119 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.259094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsbmq\" (UniqueName: \"kubernetes.io/projected/45d2feb0-2647-4011-a902-0135762b43a5-kube-api-access-qsbmq\") pod \"payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.359948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.359902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d2feb0-2647-4011-a902-0135762b43a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.360119 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.360011    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsbmq\" (UniqueName: \"kubernetes.io/projected/45d2feb0-2647-4011-a902-0135762b43a5-kube-api-access-qsbmq\") pod \"payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.360730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.360710    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d2feb0-2647-4011-a902-0135762b43a5-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.368664 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.368640    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsbmq\" (UniqueName: \"kubernetes.io/projected/45d2feb0-2647-4011-a902-0135762b43a5-kube-api-access-qsbmq\") pod \"payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") " pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.487471 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.487377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:39.759288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.759266    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"]
Sep 15 11:39:39.761098 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:39:39.761070    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d2feb0_2647_4011_a902_0135762b43a5.slice/crio-6647187b46bee9d4f7ccf8463ad46e540a14bb54a71958f73c5cedd0d3f2fe72 WatchSource:0}: Error finding container 6647187b46bee9d4f7ccf8463ad46e540a14bb54a71958f73c5cedd0d3f2fe72: Status 404 returned error can't find the container with id 6647187b46bee9d4f7ccf8463ad46e540a14bb54a71958f73c5cedd0d3f2fe72
Sep 15 11:39:39.782228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:39.782198    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2" event={"ID":"45d2feb0-2647-4011-a902-0135762b43a5","Type":"ContainerStarted","Data":"6647187b46bee9d4f7ccf8463ad46e540a14bb54a71958f73c5cedd0d3f2fe72"}
Sep 15 11:39:40.789034 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:40.789002    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2" event={"ID":"45d2feb0-2647-4011-a902-0135762b43a5","Type":"ContainerStarted","Data":"92d876d6d31a04aa2c331f1e19d047aa0fe887dea92b8de775f6d27ce83a96ff"}
Sep 15 11:39:40.789421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:40.789135    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:39:40.807029 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:40.806979    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2" podStartSLOduration=1.806961203 podStartE2EDuration="1.806961203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:39:40.806741958 +0000 UTC m=+874.584418226" watchObservedRunningTime="2026-09-15 11:39:40.806961203 +0000 UTC m=+874.584637459"
Sep 15 11:39:43.378395 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:43.378360    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:43.378863 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:43.378408    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:43.391293 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:43.391266    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:43.818384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:43.818351    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:39:51.796660 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:39:51.796627    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:40:06.632426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:06.632340    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:40:06.633529 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:06.633492    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:40:06.636938 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:06.636918    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:40:06.637925 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:06.637907    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:40:07.713581 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:07.713546    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"]
Sep 15 11:40:07.714018 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:07.713758    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2" podUID="45d2feb0-2647-4011-a902-0135762b43a5" containerName="payload-processing" containerID="cri-o://92d876d6d31a04aa2c331f1e19d047aa0fe887dea92b8de775f6d27ce83a96ff" gracePeriod=30
Sep 15 11:40:07.935449 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:07.935414    2574 generic.go:356] "Generic (PLEG): container finished" podID="45d2feb0-2647-4011-a902-0135762b43a5" containerID="92d876d6d31a04aa2c331f1e19d047aa0fe887dea92b8de775f6d27ce83a96ff" exitCode=0
Sep 15 11:40:07.935649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:07.935516    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2" event={"ID":"45d2feb0-2647-4011-a902-0135762b43a5","Type":"ContainerDied","Data":"92d876d6d31a04aa2c331f1e19d047aa0fe887dea92b8de775f6d27ce83a96ff"}
Sep 15 11:40:07.979087 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:07.979024    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:40:08.060820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.060783    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d2feb0-2647-4011-a902-0135762b43a5-plugins-config-volume\") pod \"45d2feb0-2647-4011-a902-0135762b43a5\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") "
Sep 15 11:40:08.061000 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.060831    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsbmq\" (UniqueName: \"kubernetes.io/projected/45d2feb0-2647-4011-a902-0135762b43a5-kube-api-access-qsbmq\") pod \"45d2feb0-2647-4011-a902-0135762b43a5\" (UID: \"45d2feb0-2647-4011-a902-0135762b43a5\") "
Sep 15 11:40:08.061319 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.061291    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d2feb0-2647-4011-a902-0135762b43a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45d2feb0-2647-4011-a902-0135762b43a5" (UID: "45d2feb0-2647-4011-a902-0135762b43a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:08.061699 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.061675    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d2feb0-2647-4011-a902-0135762b43a5-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:40:08.063109 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.063080    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d2feb0-2647-4011-a902-0135762b43a5-kube-api-access-qsbmq" (OuterVolumeSpecName: "kube-api-access-qsbmq") pod "45d2feb0-2647-4011-a902-0135762b43a5" (UID: "45d2feb0-2647-4011-a902-0135762b43a5"). InnerVolumeSpecName "kube-api-access-qsbmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:08.163096 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.163055    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsbmq\" (UniqueName: \"kubernetes.io/projected/45d2feb0-2647-4011-a902-0135762b43a5-kube-api-access-qsbmq\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:40:08.942462 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.942375    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"
Sep 15 11:40:08.942462 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.942420    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2" event={"ID":"45d2feb0-2647-4011-a902-0135762b43a5","Type":"ContainerDied","Data":"6647187b46bee9d4f7ccf8463ad46e540a14bb54a71958f73c5cedd0d3f2fe72"}
Sep 15 11:40:08.942973 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.942479    2574 scope.go:117] "RemoveContainer" containerID="92d876d6d31a04aa2c331f1e19d047aa0fe887dea92b8de775f6d27ce83a96ff"
Sep 15 11:40:08.958542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.958519    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"]
Sep 15 11:40:08.964989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:08.964966    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fc7a9035-547c5d7b66-svwl2"]
Sep 15 11:40:10.665722 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:10.665687    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d2feb0-2647-4011-a902-0135762b43a5" path="/var/lib/kubelet/pods/45d2feb0-2647-4011-a902-0135762b43a5/volumes"
Sep 15 11:40:14.229391 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.229345    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"]
Sep 15 11:40:14.229854 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.229839    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d2feb0-2647-4011-a902-0135762b43a5" containerName="payload-processing"
Sep 15 11:40:14.229902 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.229856    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d2feb0-2647-4011-a902-0135762b43a5" containerName="payload-processing"
Sep 15 11:40:14.229936 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.229916    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d2feb0-2647-4011-a902-0135762b43a5" containerName="payload-processing"
Sep 15 11:40:14.234459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.234443    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.236687 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.236664    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2b418982-dockercfg-jt5wm\""
Sep 15 11:40:14.237004 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.236988    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2b418982\""
Sep 15 11:40:14.244953 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.244932    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"]
Sep 15 11:40:14.323141 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.323105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng85f\" (UniqueName: \"kubernetes.io/projected/a7fd6a36-95be-4567-acb3-f34363e72198-kube-api-access-ng85f\") pod \"payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.323331 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.323166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fd6a36-95be-4567-acb3-f34363e72198-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.423930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.423884    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng85f\" (UniqueName: \"kubernetes.io/projected/a7fd6a36-95be-4567-acb3-f34363e72198-kube-api-access-ng85f\") pod \"payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.424136 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.423980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fd6a36-95be-4567-acb3-f34363e72198-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.424743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.424725    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fd6a36-95be-4567-acb3-f34363e72198-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.432813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.432784    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng85f\" (UniqueName: \"kubernetes.io/projected/a7fd6a36-95be-4567-acb3-f34363e72198-kube-api-access-ng85f\") pod \"payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.545482 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.545383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.825206 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.825179    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"]
Sep 15 11:40:14.827396 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:40:14.827369    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fd6a36_95be_4567_acb3_f34363e72198.slice/crio-e8290947d21b15015b7e0b791dcd3b0577d724b9ac5bf495999627e1b437c6eb WatchSource:0}: Error finding container e8290947d21b15015b7e0b791dcd3b0577d724b9ac5bf495999627e1b437c6eb: Status 404 returned error can't find the container with id e8290947d21b15015b7e0b791dcd3b0577d724b9ac5bf495999627e1b437c6eb
Sep 15 11:40:14.976315 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.976283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k" event={"ID":"a7fd6a36-95be-4567-acb3-f34363e72198","Type":"ContainerStarted","Data":"087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9"}
Sep 15 11:40:14.976483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.976334    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k" event={"ID":"a7fd6a36-95be-4567-acb3-f34363e72198","Type":"ContainerStarted","Data":"e8290947d21b15015b7e0b791dcd3b0577d724b9ac5bf495999627e1b437c6eb"}
Sep 15 11:40:14.976483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.976427    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:14.994101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:14.994057    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k" podStartSLOduration=0.994042961 podStartE2EDuration="994.042961ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:14.991873927 +0000 UTC m=+908.769550184" watchObservedRunningTime="2026-09-15 11:40:14.994042961 +0000 UTC m=+908.771719214"
Sep 15 11:40:25.984427 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:25.984391    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:31.118508 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.118462    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"]
Sep 15 11:40:31.122178 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.122157    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.125033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.125007    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e114ba\"/\"openshift-service-ca.crt\""
Sep 15 11:40:31.125157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.125009    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e114ba\"/\"autores-model-e114ba-kserve-self-signed-certs\""
Sep 15 11:40:31.125157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.125009    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-e114ba\"/\"default-dockercfg-bsknt\""
Sep 15 11:40:31.125157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.125054    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-e114ba\"/\"kube-root-ca.crt\""
Sep 15 11:40:31.133604 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.133583    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"]
Sep 15 11:40:31.293710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.293675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-home\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.293875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.293729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-model-cache\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.293875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.293787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-tmp-dir\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.293875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.293830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtm9l\" (UniqueName: \"kubernetes.io/projected/196ea361-afaf-49c8-a842-f3c3c96acccc-kube-api-access-dtm9l\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.294014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.293881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196ea361-afaf-49c8-a842-f3c3c96acccc-tls-certs\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.294014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.293934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-dshm\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.394637 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-home\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.394796 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394658    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-model-cache\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.394796 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-tmp-dir\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.394915 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtm9l\" (UniqueName: \"kubernetes.io/projected/196ea361-afaf-49c8-a842-f3c3c96acccc-kube-api-access-dtm9l\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.394915 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394862    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196ea361-afaf-49c8-a842-f3c3c96acccc-tls-certs\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.394915 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-dshm\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.395075 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.394999    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-model-cache\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.395075 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.395027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-home\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.395186 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.395126    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-tmp-dir\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.397271 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.397246    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-dshm\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.397481 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.397464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196ea361-afaf-49c8-a842-f3c3c96acccc-tls-certs\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.402710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.402688    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtm9l\" (UniqueName: \"kubernetes.io/projected/196ea361-afaf-49c8-a842-f3c3c96acccc-kube-api-access-dtm9l\") pod \"autores-model-e114ba-kserve-dd454798c-q6xxz\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") " pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.432756 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.432732    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:31.703669 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:31.703638    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"]
Sep 15 11:40:31.705446 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:40:31.705415    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod196ea361_afaf_49c8_a842_f3c3c96acccc.slice/crio-76b389d1839ac416b5867399a010e458d16ff5a56c41bdc1e9dc441837da4cf1 WatchSource:0}: Error finding container 76b389d1839ac416b5867399a010e458d16ff5a56c41bdc1e9dc441837da4cf1: Status 404 returned error can't find the container with id 76b389d1839ac416b5867399a010e458d16ff5a56c41bdc1e9dc441837da4cf1
Sep 15 11:40:32.068392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:32.068298    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz" event={"ID":"196ea361-afaf-49c8-a842-f3c3c96acccc","Type":"ContainerStarted","Data":"4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132"}
Sep 15 11:40:32.068392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:32.068344    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz" event={"ID":"196ea361-afaf-49c8-a842-f3c3c96acccc","Type":"ContainerStarted","Data":"76b389d1839ac416b5867399a010e458d16ff5a56c41bdc1e9dc441837da4cf1"}
Sep 15 11:40:32.085160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:32.085114    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz" podStartSLOduration=1.08509817 podStartE2EDuration="1.08509817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:32.083189042 +0000 UTC m=+925.860865298" watchObservedRunningTime="2026-09-15 11:40:32.08509817 +0000 UTC m=+925.862774427"
Sep 15 11:40:41.433731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:41.433684    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:41.434242 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:41.433755    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:41.446593 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:41.446567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:42.132663 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.132633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:40:42.475729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.475691    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"]
Sep 15 11:40:42.476226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.475945    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k" podUID="a7fd6a36-95be-4567-acb3-f34363e72198" containerName="payload-pre-processing" containerID="cri-o://087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9" gracePeriod=30
Sep 15 11:40:42.731521 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.731439    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:42.795411 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.795382    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng85f\" (UniqueName: \"kubernetes.io/projected/a7fd6a36-95be-4567-acb3-f34363e72198-kube-api-access-ng85f\") pod \"a7fd6a36-95be-4567-acb3-f34363e72198\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") "
Sep 15 11:40:42.795596 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.795418    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fd6a36-95be-4567-acb3-f34363e72198-plugins-config-volume\") pod \"a7fd6a36-95be-4567-acb3-f34363e72198\" (UID: \"a7fd6a36-95be-4567-acb3-f34363e72198\") "
Sep 15 11:40:42.795865 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.795831    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fd6a36-95be-4567-acb3-f34363e72198-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7fd6a36-95be-4567-acb3-f34363e72198" (UID: "a7fd6a36-95be-4567-acb3-f34363e72198"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:40:42.797473 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.797442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fd6a36-95be-4567-acb3-f34363e72198-kube-api-access-ng85f" (OuterVolumeSpecName: "kube-api-access-ng85f") pod "a7fd6a36-95be-4567-acb3-f34363e72198" (UID: "a7fd6a36-95be-4567-acb3-f34363e72198"). InnerVolumeSpecName "kube-api-access-ng85f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:40:42.896427 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.896379    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng85f\" (UniqueName: \"kubernetes.io/projected/a7fd6a36-95be-4567-acb3-f34363e72198-kube-api-access-ng85f\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:40:42.896427 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:42.896419    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7fd6a36-95be-4567-acb3-f34363e72198-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:40:43.133007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.132913    2574 generic.go:356] "Generic (PLEG): container finished" podID="a7fd6a36-95be-4567-acb3-f34363e72198" containerID="087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9" exitCode=0
Sep 15 11:40:43.133007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.132985    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"
Sep 15 11:40:43.133209 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.133000    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k" event={"ID":"a7fd6a36-95be-4567-acb3-f34363e72198","Type":"ContainerDied","Data":"087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9"}
Sep 15 11:40:43.133209 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.133090    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k" event={"ID":"a7fd6a36-95be-4567-acb3-f34363e72198","Type":"ContainerDied","Data":"e8290947d21b15015b7e0b791dcd3b0577d724b9ac5bf495999627e1b437c6eb"}
Sep 15 11:40:43.133209 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.133113    2574 scope.go:117] "RemoveContainer" containerID="087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9"
Sep 15 11:40:43.142601 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.142581    2574 scope.go:117] "RemoveContainer" containerID="087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9"
Sep 15 11:40:43.142905 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:40:43.142886    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9\": container with ID starting with 087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9 not found: ID does not exist" containerID="087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9"
Sep 15 11:40:43.142958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.142915    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9"} err="failed to get container status \"087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9\": rpc error: code = NotFound desc = could not find container \"087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9\": container with ID starting with 087e0e080dad30525dc06570813c1c83306e27c92f9a8ff0229afc55570021e9 not found: ID does not exist"
Sep 15 11:40:43.154864 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.154838    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"]
Sep 15 11:40:43.160860 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:43.160838    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2b418982-55c46d788b-wt27k"]
Sep 15 11:40:44.666451 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:44.666419    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fd6a36-95be-4567-acb3-f34363e72198" path="/var/lib/kubelet/pods/a7fd6a36-95be-4567-acb3-f34363e72198/volumes"
Sep 15 11:40:52.193314 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.193284    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"]
Sep 15 11:40:52.195673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.193761    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fd6a36-95be-4567-acb3-f34363e72198" containerName="payload-pre-processing"
Sep 15 11:40:52.195673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.193775    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fd6a36-95be-4567-acb3-f34363e72198" containerName="payload-pre-processing"
Sep 15 11:40:52.195673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.193843    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fd6a36-95be-4567-acb3-f34363e72198" containerName="payload-pre-processing"
Sep 15 11:40:52.196825 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.196810    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.200750 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.200730    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ed7d9f67\""
Sep 15 11:40:52.200896 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.200795    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ed7d9f67-dockercfg-4n9t9\""
Sep 15 11:40:52.213770 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.213748    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"]
Sep 15 11:40:52.289579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.289550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-plugins-config-volume\") pod \"payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") " pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.289755 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.289603    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rjlf\" (UniqueName: \"kubernetes.io/projected/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-kube-api-access-8rjlf\") pod \"payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") " pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.390563 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.390521    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-plugins-config-volume\") pod \"payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") " pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.390725 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.390585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rjlf\" (UniqueName: \"kubernetes.io/projected/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-kube-api-access-8rjlf\") pod \"payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") " pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.391232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.391212    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-plugins-config-volume\") pod \"payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") " pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.403057 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.403032    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rjlf\" (UniqueName: \"kubernetes.io/projected/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-kube-api-access-8rjlf\") pod \"payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") " pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.507026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.506934    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:52.777766 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:52.777741    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"]
Sep 15 11:40:52.779062 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:40:52.779032    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4f1bf6_db39_481c_b8b0_ee5368c1fd32.slice/crio-7a34634cfcf297e23621f4c6747c5cb06334f8ada3f5aa93c4df720f1aa001c9 WatchSource:0}: Error finding container 7a34634cfcf297e23621f4c6747c5cb06334f8ada3f5aa93c4df720f1aa001c9: Status 404 returned error can't find the container with id 7a34634cfcf297e23621f4c6747c5cb06334f8ada3f5aa93c4df720f1aa001c9
Sep 15 11:40:53.189555 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:53.189479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b" event={"ID":"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32","Type":"ContainerStarted","Data":"6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26"}
Sep 15 11:40:53.189754 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:53.189571    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b" event={"ID":"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32","Type":"ContainerStarted","Data":"7a34634cfcf297e23621f4c6747c5cb06334f8ada3f5aa93c4df720f1aa001c9"}
Sep 15 11:40:53.189754 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:53.189591    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:40:53.207748 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:40:53.207698    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b" podStartSLOduration=1.207681247 podStartE2EDuration="1.207681247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:40:53.20582291 +0000 UTC m=+946.983499168" watchObservedRunningTime="2026-09-15 11:40:53.207681247 +0000 UTC m=+946.985357503"
Sep 15 11:41:04.197341 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:04.197311    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:41:04.572707 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:04.572625    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"]
Sep 15 11:41:17.197612 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.197575    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"]
Sep 15 11:41:17.198122 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.197778    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27" podUID="ab8e0b33-563f-4827-bfc2-14ddbdc53438" containerName="payload-processing" containerID="cri-o://e2977e5c6d22f0457c4f8282e9d42f7e3a74a21251b26462a04d7bdfbb725ce7" gracePeriod=30
Sep 15 11:41:17.318571 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.318538    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27" event={"ID":"ab8e0b33-563f-4827-bfc2-14ddbdc53438","Type":"ContainerDied","Data":"e2977e5c6d22f0457c4f8282e9d42f7e3a74a21251b26462a04d7bdfbb725ce7"}
Sep 15 11:41:17.318700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.318491    2574 generic.go:356] "Generic (PLEG): container finished" podID="ab8e0b33-563f-4827-bfc2-14ddbdc53438" containerID="e2977e5c6d22f0457c4f8282e9d42f7e3a74a21251b26462a04d7bdfbb725ce7" exitCode=0
Sep 15 11:41:17.459201 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.459141    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:41:17.645672 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.645639    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8c8n\" (UniqueName: \"kubernetes.io/projected/ab8e0b33-563f-4827-bfc2-14ddbdc53438-kube-api-access-f8c8n\") pod \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") "
Sep 15 11:41:17.645865 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.645681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e0b33-563f-4827-bfc2-14ddbdc53438-plugins-config-volume\") pod \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\" (UID: \"ab8e0b33-563f-4827-bfc2-14ddbdc53438\") "
Sep 15 11:41:17.646070 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.646047    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab8e0b33-563f-4827-bfc2-14ddbdc53438-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ab8e0b33-563f-4827-bfc2-14ddbdc53438" (UID: "ab8e0b33-563f-4827-bfc2-14ddbdc53438"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:17.647733 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.647711    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8e0b33-563f-4827-bfc2-14ddbdc53438-kube-api-access-f8c8n" (OuterVolumeSpecName: "kube-api-access-f8c8n") pod "ab8e0b33-563f-4827-bfc2-14ddbdc53438" (UID: "ab8e0b33-563f-4827-bfc2-14ddbdc53438"). InnerVolumeSpecName "kube-api-access-f8c8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:17.747210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.747120    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8c8n\" (UniqueName: \"kubernetes.io/projected/ab8e0b33-563f-4827-bfc2-14ddbdc53438-kube-api-access-f8c8n\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:17.747210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:17.747154    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab8e0b33-563f-4827-bfc2-14ddbdc53438-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:18.325568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:18.325542    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"
Sep 15 11:41:18.325568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:18.325558    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27" event={"ID":"ab8e0b33-563f-4827-bfc2-14ddbdc53438","Type":"ContainerDied","Data":"f1bde4c24643381d7c440bca63f8315063353714b44ffce6299bd10cc9c5bbe7"}
Sep 15 11:41:18.326075 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:18.325615    2574 scope.go:117] "RemoveContainer" containerID="e2977e5c6d22f0457c4f8282e9d42f7e3a74a21251b26462a04d7bdfbb725ce7"
Sep 15 11:41:18.348135 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:18.348109    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"]
Sep 15 11:41:18.356594 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:18.356570    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-e114ba-7dcd6db77-z2v27"]
Sep 15 11:41:18.667434 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:18.667386    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab8e0b33-563f-4827-bfc2-14ddbdc53438" path="/var/lib/kubelet/pods/ab8e0b33-563f-4827-bfc2-14ddbdc53438/volumes"
Sep 15 11:41:19.333464 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.333430    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"]
Sep 15 11:41:19.333972 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.333758    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b" podUID="fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" containerName="payload-processing" containerID="cri-o://6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26" gracePeriod=30
Sep 15 11:41:19.589010 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.588943    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:41:19.592210 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.592174    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz" podUID="196ea361-afaf-49c8-a842-f3c3c96acccc" containerName="main" containerID="cri-o://4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132" gracePeriod=45
Sep 15 11:41:19.766014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.765979    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rjlf\" (UniqueName: \"kubernetes.io/projected/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-kube-api-access-8rjlf\") pod \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") "
Sep 15 11:41:19.766156 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.766142    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-plugins-config-volume\") pod \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\" (UID: \"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32\") "
Sep 15 11:41:19.766671 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.766637    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" (UID: "fe4f1bf6-db39-481c-b8b0-ee5368c1fd32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:41:19.769636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.769607    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-kube-api-access-8rjlf" (OuterVolumeSpecName: "kube-api-access-8rjlf") pod "fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" (UID: "fe4f1bf6-db39-481c-b8b0-ee5368c1fd32"). InnerVolumeSpecName "kube-api-access-8rjlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:19.828329 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.828306    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:41:19.867005 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.866931    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-home\") pod \"196ea361-afaf-49c8-a842-f3c3c96acccc\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") "
Sep 15 11:41:19.867005 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.866996    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtm9l\" (UniqueName: \"kubernetes.io/projected/196ea361-afaf-49c8-a842-f3c3c96acccc-kube-api-access-dtm9l\") pod \"196ea361-afaf-49c8-a842-f3c3c96acccc\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") "
Sep 15 11:41:19.867220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867082    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-model-cache\") pod \"196ea361-afaf-49c8-a842-f3c3c96acccc\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") "
Sep 15 11:41:19.867220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867125    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-tmp-dir\") pod \"196ea361-afaf-49c8-a842-f3c3c96acccc\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") "
Sep 15 11:41:19.867220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867170    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196ea361-afaf-49c8-a842-f3c3c96acccc-tls-certs\") pod \"196ea361-afaf-49c8-a842-f3c3c96acccc\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") "
Sep 15 11:41:19.867382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867218    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-home" (OuterVolumeSpecName: "home") pod "196ea361-afaf-49c8-a842-f3c3c96acccc" (UID: "196ea361-afaf-49c8-a842-f3c3c96acccc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:41:19.867457 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867404    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-model-cache" (OuterVolumeSpecName: "model-cache") pod "196ea361-afaf-49c8-a842-f3c3c96acccc" (UID: "196ea361-afaf-49c8-a842-f3c3c96acccc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:41:19.867457 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867420    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "196ea361-afaf-49c8-a842-f3c3c96acccc" (UID: "196ea361-afaf-49c8-a842-f3c3c96acccc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:41:19.867587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867469    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-dshm\") pod \"196ea361-afaf-49c8-a842-f3c3c96acccc\" (UID: \"196ea361-afaf-49c8-a842-f3c3c96acccc\") "
Sep 15 11:41:19.867859 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867836    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.867983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867865    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.867983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867880    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.867983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867894    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rjlf\" (UniqueName: \"kubernetes.io/projected/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32-kube-api-access-8rjlf\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.867983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.867915    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.871087 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.871033    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/196ea361-afaf-49c8-a842-f3c3c96acccc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "196ea361-afaf-49c8-a842-f3c3c96acccc" (UID: "196ea361-afaf-49c8-a842-f3c3c96acccc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:41:19.872317 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.872293    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196ea361-afaf-49c8-a842-f3c3c96acccc-kube-api-access-dtm9l" (OuterVolumeSpecName: "kube-api-access-dtm9l") pod "196ea361-afaf-49c8-a842-f3c3c96acccc" (UID: "196ea361-afaf-49c8-a842-f3c3c96acccc"). InnerVolumeSpecName "kube-api-access-dtm9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:41:19.872437 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.872402    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-dshm" (OuterVolumeSpecName: "dshm") pod "196ea361-afaf-49c8-a842-f3c3c96acccc" (UID: "196ea361-afaf-49c8-a842-f3c3c96acccc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:41:19.969187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.969158    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtm9l\" (UniqueName: \"kubernetes.io/projected/196ea361-afaf-49c8-a842-f3c3c96acccc-kube-api-access-dtm9l\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.969187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.969185    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/196ea361-afaf-49c8-a842-f3c3c96acccc-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:19.969187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:19.969196    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/196ea361-afaf-49c8-a842-f3c3c96acccc-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:41:20.342551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.342518    2574 generic.go:356] "Generic (PLEG): container finished" podID="196ea361-afaf-49c8-a842-f3c3c96acccc" containerID="4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132" exitCode=0
Sep 15 11:41:20.342978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.342570    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"
Sep 15 11:41:20.342978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.342607    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz" event={"ID":"196ea361-afaf-49c8-a842-f3c3c96acccc","Type":"ContainerDied","Data":"4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132"}
Sep 15 11:41:20.342978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.342653    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz" event={"ID":"196ea361-afaf-49c8-a842-f3c3c96acccc","Type":"ContainerDied","Data":"76b389d1839ac416b5867399a010e458d16ff5a56c41bdc1e9dc441837da4cf1"}
Sep 15 11:41:20.342978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.342674    2574 scope.go:117] "RemoveContainer" containerID="4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132"
Sep 15 11:41:20.344065 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.344024    2574 generic.go:356] "Generic (PLEG): container finished" podID="fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" containerID="6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26" exitCode=0
Sep 15 11:41:20.344138 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.344072    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"
Sep 15 11:41:20.344138 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.344119    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b" event={"ID":"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32","Type":"ContainerDied","Data":"6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26"}
Sep 15 11:41:20.344254 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.344165    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b" event={"ID":"fe4f1bf6-db39-481c-b8b0-ee5368c1fd32","Type":"ContainerDied","Data":"7a34634cfcf297e23621f4c6747c5cb06334f8ada3f5aa93c4df720f1aa001c9"}
Sep 15 11:41:20.363765 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.363748    2574 scope.go:117] "RemoveContainer" containerID="4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132"
Sep 15 11:41:20.364066 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:41:20.364045    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132\": container with ID starting with 4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132 not found: ID does not exist" containerID="4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132"
Sep 15 11:41:20.364120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.364075    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132"} err="failed to get container status \"4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132\": rpc error: code = NotFound desc = could not find container \"4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132\": container with ID starting with 4fedf5994eacea5a0b5a5fb485b746f532af9bce177d3fb618da3d151a701132 not found: ID does not exist"
Sep 15 11:41:20.364120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.364092    2574 scope.go:117] "RemoveContainer" containerID="6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26"
Sep 15 11:41:20.375819 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.375800    2574 scope.go:117] "RemoveContainer" containerID="6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26"
Sep 15 11:41:20.375888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.375803    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"]
Sep 15 11:41:20.376098 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:41:20.376082    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26\": container with ID starting with 6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26 not found: ID does not exist" containerID="6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26"
Sep 15 11:41:20.376141 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.376104    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26"} err="failed to get container status \"6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26\": rpc error: code = NotFound desc = could not find container \"6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26\": container with ID starting with 6eb896f9aed3c9f4e4946627990e2d01e0fc0d7889d2985a7d5c359f4b97db26 not found: ID does not exist"
Sep 15 11:41:20.379847 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.379827    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-e114ba/autores-model-e114ba-kserve-dd454798c-q6xxz"]
Sep 15 11:41:20.391049 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.391026    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"]
Sep 15 11:41:20.395047 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.395024    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ed7d9f67-775558d5d5-csh5b"]
Sep 15 11:41:20.665770 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.665740    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="196ea361-afaf-49c8-a842-f3c3c96acccc" path="/var/lib/kubelet/pods/196ea361-afaf-49c8-a842-f3c3c96acccc/volumes"
Sep 15 11:41:20.666175 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:20.666158    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" path="/var/lib/kubelet/pods/fe4f1bf6-db39-481c-b8b0-ee5368c1fd32/volumes"
Sep 15 11:41:32.647673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.647594    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"]
Sep 15 11:41:32.648031 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648019    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" containerName="payload-processing"
Sep 15 11:41:32.648078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648034    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" containerName="payload-processing"
Sep 15 11:41:32.648078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648049    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab8e0b33-563f-4827-bfc2-14ddbdc53438" containerName="payload-processing"
Sep 15 11:41:32.648078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648056    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab8e0b33-563f-4827-bfc2-14ddbdc53438" containerName="payload-processing"
Sep 15 11:41:32.648078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648074    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="196ea361-afaf-49c8-a842-f3c3c96acccc" containerName="main"
Sep 15 11:41:32.648078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648080    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="196ea361-afaf-49c8-a842-f3c3c96acccc" containerName="main"
Sep 15 11:41:32.648231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648145    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="196ea361-afaf-49c8-a842-f3c3c96acccc" containerName="main"
Sep 15 11:41:32.648231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648151    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4f1bf6-db39-481c-b8b0-ee5368c1fd32" containerName="payload-processing"
Sep 15 11:41:32.648231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.648157    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab8e0b33-563f-4827-bfc2-14ddbdc53438" containerName="payload-processing"
Sep 15 11:41:32.652621 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.652602    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.655101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.655077    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a1c362-dockercfg-8xpr6\""
Sep 15 11:41:32.655617 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.655597    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a1c362\""
Sep 15 11:41:32.666693 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.666669    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"]
Sep 15 11:41:32.691012 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.690984    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.691157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.691039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cvqg\" (UniqueName: \"kubernetes.io/projected/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-kube-api-access-7cvqg\") pod \"payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.791941 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.791894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cvqg\" (UniqueName: \"kubernetes.io/projected/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-kube-api-access-7cvqg\") pod \"payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.792096 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.792084    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.792793 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.792773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.799982 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.799949    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cvqg\" (UniqueName: \"kubernetes.io/projected/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-kube-api-access-7cvqg\") pod \"payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") " pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:32.963554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:32.963450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:33.235108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:33.235078    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"]
Sep 15 11:41:33.237066 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:41:33.237038    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7599c6_6c7f_4656_a6bf_5f04a62a1bb4.slice/crio-a27fce43270e3ef9d6a5bd7e32d1c37d9a1a5c2cbbb9475bbdfcb6cd48359986 WatchSource:0}: Error finding container a27fce43270e3ef9d6a5bd7e32d1c37d9a1a5c2cbbb9475bbdfcb6cd48359986: Status 404 returned error can't find the container with id a27fce43270e3ef9d6a5bd7e32d1c37d9a1a5c2cbbb9475bbdfcb6cd48359986
Sep 15 11:41:33.418065 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:33.418030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6" event={"ID":"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4","Type":"ContainerStarted","Data":"33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e"}
Sep 15 11:41:33.418273 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:33.418085    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6" event={"ID":"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4","Type":"ContainerStarted","Data":"a27fce43270e3ef9d6a5bd7e32d1c37d9a1a5c2cbbb9475bbdfcb6cd48359986"}
Sep 15 11:41:33.418273 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:33.418156    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:33.435914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:33.435859    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6" podStartSLOduration=1.435842374 podStartE2EDuration="1.435842374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:41:33.434288609 +0000 UTC m=+987.211964880" watchObservedRunningTime="2026-09-15 11:41:33.435842374 +0000 UTC m=+987.213518696"
Sep 15 11:41:38.758355 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.758323    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"]
Sep 15 11:41:38.761835 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.761815    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.764046 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.764027    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"premium-w4-2fab1f-kserve-self-signed-certs\""
Sep 15 11:41:38.771536 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.771487    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"]
Sep 15 11:41:38.852709 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.852677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-dshm\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.852709 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.852716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-home\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.852920 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.852758    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-model-cache\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.852920 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.852832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-229zz\" (UniqueName: \"kubernetes.io/projected/ce1b92fe-d45f-4883-86b1-974250bc389c-kube-api-access-229zz\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.852920 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.852881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-tmp-dir\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.853017 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.852942    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1b92fe-d45f-4883-86b1-974250bc389c-tls-certs\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.953774 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.953741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-dshm\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.953774 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.953781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-home\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.953817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-model-cache\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.953860    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-229zz\" (UniqueName: \"kubernetes.io/projected/ce1b92fe-d45f-4883-86b1-974250bc389c-kube-api-access-229zz\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.953890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-tmp-dir\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.953951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1b92fe-d45f-4883-86b1-974250bc389c-tls-certs\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954259 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.954221    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-home\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954259 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.954251    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-model-cache\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.954347 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.954309    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-tmp-dir\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.956031 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.956010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-dshm\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.956270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.956254    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1b92fe-d45f-4883-86b1-974250bc389c-tls-certs\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:38.961905 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:38.961882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-229zz\" (UniqueName: \"kubernetes.io/projected/ce1b92fe-d45f-4883-86b1-974250bc389c-kube-api-access-229zz\") pod \"premium-w4-2fab1f-kserve-68d65bc648-dffr9\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") " pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:39.071820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:39.071738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:39.344151 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:39.344122    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"]
Sep 15 11:41:39.454655 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:39.454621    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9" event={"ID":"ce1b92fe-d45f-4883-86b1-974250bc389c","Type":"ContainerStarted","Data":"5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295"}
Sep 15 11:41:39.454655 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:39.454667    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9" event={"ID":"ce1b92fe-d45f-4883-86b1-974250bc389c","Type":"ContainerStarted","Data":"3af7515d894042f6c8e6edbd604af661d531244fe1e15f96198fa22239a944e2"}
Sep 15 11:41:39.471934 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:39.471875    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9" podStartSLOduration=1.4718590360000001 podStartE2EDuration="1.471859036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:41:39.470602119 +0000 UTC m=+993.248278378" watchObservedRunningTime="2026-09-15 11:41:39.471859036 +0000 UTC m=+993.249535293"
Sep 15 11:41:44.425713 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:44.425681    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:41:49.072257 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:49.072218    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:49.072802 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:49.072289    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:49.084917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:49.084890    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:41:49.524058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:41:49.524031    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:43:04.432371 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.432289    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"]
Sep 15 11:43:04.432863 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.432560    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6" podUID="bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" containerName="payload-processing" containerID="cri-o://33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e" gracePeriod=30
Sep 15 11:43:04.701587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.701556    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:43:04.796736 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.796691    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cvqg\" (UniqueName: \"kubernetes.io/projected/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-kube-api-access-7cvqg\") pod \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") "
Sep 15 11:43:04.796933 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.796823    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-plugins-config-volume\") pod \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\" (UID: \"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4\") "
Sep 15 11:43:04.797188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.797155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" (UID: "bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:04.798889 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.798865    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-kube-api-access-7cvqg" (OuterVolumeSpecName: "kube-api-access-7cvqg") pod "bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" (UID: "bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4"). InnerVolumeSpecName "kube-api-access-7cvqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:04.898384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.898352    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cvqg\" (UniqueName: \"kubernetes.io/projected/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-kube-api-access-7cvqg\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:43:04.898384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.898381    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:43:04.924078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.924045    2574 generic.go:356] "Generic (PLEG): container finished" podID="bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" containerID="33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e" exitCode=0
Sep 15 11:43:04.924240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.924088    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"
Sep 15 11:43:04.924240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.924134    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6" event={"ID":"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4","Type":"ContainerDied","Data":"33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e"}
Sep 15 11:43:04.924240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.924195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6" event={"ID":"bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4","Type":"ContainerDied","Data":"a27fce43270e3ef9d6a5bd7e32d1c37d9a1a5c2cbbb9475bbdfcb6cd48359986"}
Sep 15 11:43:04.924240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.924223    2574 scope.go:117] "RemoveContainer" containerID="33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e"
Sep 15 11:43:04.933958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.933937    2574 scope.go:117] "RemoveContainer" containerID="33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e"
Sep 15 11:43:04.934264 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:43:04.934246    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e\": container with ID starting with 33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e not found: ID does not exist" containerID="33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e"
Sep 15 11:43:04.934334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.934276    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e"} err="failed to get container status \"33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e\": rpc error: code = NotFound desc = could not find container \"33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e\": container with ID starting with 33f4b7207d5662d0dda2e81625765cd09b9ba5c0029b045b53cd13361831559e not found: ID does not exist"
Sep 15 11:43:04.945967 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.945916    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"]
Sep 15 11:43:04.949994 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:04.949974    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a1c362-d47ff4df-bb7d6"]
Sep 15 11:43:06.295352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.295319    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"]
Sep 15 11:43:06.295797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.295782    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" containerName="payload-processing"
Sep 15 11:43:06.295797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.295798    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" containerName="payload-processing"
Sep 15 11:43:06.295880 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.295874    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" containerName="payload-processing"
Sep 15 11:43:06.300184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.300169    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.303753 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.303732    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-53d3db96\""
Sep 15 11:43:06.317416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.317384    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"]
Sep 15 11:43:06.331437 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.331413    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"]
Sep 15 11:43:06.335782 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.335763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.338091 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.338066    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-53d3db96-dockercfg-7ftx5\""
Sep 15 11:43:06.354325 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.354292    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"]
Sep 15 11:43:06.412906 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.412869    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a32eccb-086b-4155-bb04-fe0a6d476daa-plugins-config-volume\") pod \"payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") " pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.413075 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.412912    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frjzc\" (UniqueName: \"kubernetes.io/projected/6b6f24bc-1abf-401a-a191-13435ba902b6-kube-api-access-frjzc\") pod \"payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.413075 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.412988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f24bc-1abf-401a-a191-13435ba902b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.413151 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.413074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ltw8\" (UniqueName: \"kubernetes.io/projected/6a32eccb-086b-4155-bb04-fe0a6d476daa-kube-api-access-4ltw8\") pod \"payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") " pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.514527 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.514464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a32eccb-086b-4155-bb04-fe0a6d476daa-plugins-config-volume\") pod \"payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") " pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.514702 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.514536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frjzc\" (UniqueName: \"kubernetes.io/projected/6b6f24bc-1abf-401a-a191-13435ba902b6-kube-api-access-frjzc\") pod \"payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.514702 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.514567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f24bc-1abf-401a-a191-13435ba902b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.514702 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.514602    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ltw8\" (UniqueName: \"kubernetes.io/projected/6a32eccb-086b-4155-bb04-fe0a6d476daa-kube-api-access-4ltw8\") pod \"payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") " pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.515192 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.515171    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a32eccb-086b-4155-bb04-fe0a6d476daa-plugins-config-volume\") pod \"payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") " pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.515262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.515244    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f24bc-1abf-401a-a191-13435ba902b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.524312 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.524278    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ltw8\" (UniqueName: \"kubernetes.io/projected/6a32eccb-086b-4155-bb04-fe0a6d476daa-kube-api-access-4ltw8\") pod \"payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") " pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.524438 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.524357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frjzc\" (UniqueName: \"kubernetes.io/projected/6b6f24bc-1abf-401a-a191-13435ba902b6-kube-api-access-frjzc\") pod \"payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.610801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.610714    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:06.649612 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.649579    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-53d3db96-dockercfg-7ftx5\""
Sep 15 11:43:06.657532 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.657476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:06.672710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.671904    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4" path="/var/lib/kubelet/pods/bf7599c6-6c7f-4656-a6bf-5f04a62a1bb4/volumes"
Sep 15 11:43:06.891005 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.890980    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"]
Sep 15 11:43:06.892925 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:43:06.892894    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6f24bc_1abf_401a_a191_13435ba902b6.slice/crio-8cc83f2680bc613c8501437b2192006a4b0ca2b9d80305098ebc6b9cc1562b90 WatchSource:0}: Error finding container 8cc83f2680bc613c8501437b2192006a4b0ca2b9d80305098ebc6b9cc1562b90: Status 404 returned error can't find the container with id 8cc83f2680bc613c8501437b2192006a4b0ca2b9d80305098ebc6b9cc1562b90
Sep 15 11:43:06.937550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.937524    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82" event={"ID":"6b6f24bc-1abf-401a-a191-13435ba902b6","Type":"ContainerStarted","Data":"8cc83f2680bc613c8501437b2192006a4b0ca2b9d80305098ebc6b9cc1562b90"}
Sep 15 11:43:06.985237 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:06.985211    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"]
Sep 15 11:43:06.986757 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:43:06.986726    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a32eccb_086b_4155_bb04_fe0a6d476daa.slice/crio-1b81ec98a0e3fc761d6e3e9958738692a9a47af942e3a08dbf1d12d99620b757 WatchSource:0}: Error finding container 1b81ec98a0e3fc761d6e3e9958738692a9a47af942e3a08dbf1d12d99620b757: Status 404 returned error can't find the container with id 1b81ec98a0e3fc761d6e3e9958738692a9a47af942e3a08dbf1d12d99620b757
Sep 15 11:43:07.943720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.943687    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82" event={"ID":"6b6f24bc-1abf-401a-a191-13435ba902b6","Type":"ContainerStarted","Data":"69a2cace135bd6ccc0a0a4f5666a0fbd3d4ec34aed3befca82187fb3648bdaf6"}
Sep 15 11:43:07.944161 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.943761    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:07.945393 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.945367    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m" event={"ID":"6a32eccb-086b-4155-bb04-fe0a6d476daa","Type":"ContainerStarted","Data":"968cb58c6d88aae32110c8682be3819282087877bb7f290c599893de64f66229"}
Sep 15 11:43:07.945486 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.945411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m" event={"ID":"6a32eccb-086b-4155-bb04-fe0a6d476daa","Type":"ContainerStarted","Data":"1b81ec98a0e3fc761d6e3e9958738692a9a47af942e3a08dbf1d12d99620b757"}
Sep 15 11:43:07.945486 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.945434    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:07.964781 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.964742    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82" podStartSLOduration=1.964727421 podStartE2EDuration="1.964727421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:07.96260837 +0000 UTC m=+1081.740284628" watchObservedRunningTime="2026-09-15 11:43:07.964727421 +0000 UTC m=+1081.742403678"
Sep 15 11:43:07.980930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:07.980880    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m" podStartSLOduration=1.9808607569999999 podStartE2EDuration="1.980860757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:43:07.979316115 +0000 UTC m=+1081.756992373" watchObservedRunningTime="2026-09-15 11:43:07.980860757 +0000 UTC m=+1081.758537016"
Sep 15 11:43:18.952515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:18.952468    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:18.952984 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:18.952579    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:49.992460 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:49.992427    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"]
Sep 15 11:43:49.992981 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:49.992729    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m" podUID="6a32eccb-086b-4155-bb04-fe0a6d476daa" containerName="payload-processing" containerID="cri-o://968cb58c6d88aae32110c8682be3819282087877bb7f290c599893de64f66229" gracePeriod=30
Sep 15 11:43:50.003893 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.002296    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"]
Sep 15 11:43:50.003893 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.002547    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82" podUID="6b6f24bc-1abf-401a-a191-13435ba902b6" containerName="payload-pre-processing" containerID="cri-o://69a2cace135bd6ccc0a0a4f5666a0fbd3d4ec34aed3befca82187fb3648bdaf6" gracePeriod=30
Sep 15 11:43:50.172094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.172063    2574 generic.go:356] "Generic (PLEG): container finished" podID="6a32eccb-086b-4155-bb04-fe0a6d476daa" containerID="968cb58c6d88aae32110c8682be3819282087877bb7f290c599893de64f66229" exitCode=0
Sep 15 11:43:50.172428 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.172299    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m" event={"ID":"6a32eccb-086b-4155-bb04-fe0a6d476daa","Type":"ContainerDied","Data":"968cb58c6d88aae32110c8682be3819282087877bb7f290c599893de64f66229"}
Sep 15 11:43:50.174942 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.174914    2574 generic.go:356] "Generic (PLEG): container finished" podID="6b6f24bc-1abf-401a-a191-13435ba902b6" containerID="69a2cace135bd6ccc0a0a4f5666a0fbd3d4ec34aed3befca82187fb3648bdaf6" exitCode=0
Sep 15 11:43:50.175072 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.174962    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82" event={"ID":"6b6f24bc-1abf-401a-a191-13435ba902b6","Type":"ContainerDied","Data":"69a2cace135bd6ccc0a0a4f5666a0fbd3d4ec34aed3befca82187fb3648bdaf6"}
Sep 15 11:43:50.289736 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.289709    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:50.293058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.293037    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:50.362822 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.362794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f24bc-1abf-401a-a191-13435ba902b6-plugins-config-volume\") pod \"6b6f24bc-1abf-401a-a191-13435ba902b6\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") "
Sep 15 11:43:50.362992 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.362835    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a32eccb-086b-4155-bb04-fe0a6d476daa-plugins-config-volume\") pod \"6a32eccb-086b-4155-bb04-fe0a6d476daa\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") "
Sep 15 11:43:50.362992 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.362919    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ltw8\" (UniqueName: \"kubernetes.io/projected/6a32eccb-086b-4155-bb04-fe0a6d476daa-kube-api-access-4ltw8\") pod \"6a32eccb-086b-4155-bb04-fe0a6d476daa\" (UID: \"6a32eccb-086b-4155-bb04-fe0a6d476daa\") "
Sep 15 11:43:50.362992 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.362951    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frjzc\" (UniqueName: \"kubernetes.io/projected/6b6f24bc-1abf-401a-a191-13435ba902b6-kube-api-access-frjzc\") pod \"6b6f24bc-1abf-401a-a191-13435ba902b6\" (UID: \"6b6f24bc-1abf-401a-a191-13435ba902b6\") "
Sep 15 11:43:50.363197 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.363171    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6f24bc-1abf-401a-a191-13435ba902b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b6f24bc-1abf-401a-a191-13435ba902b6" (UID: "6b6f24bc-1abf-401a-a191-13435ba902b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:50.363313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.363292    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a32eccb-086b-4155-bb04-fe0a6d476daa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a32eccb-086b-4155-bb04-fe0a6d476daa" (UID: "6a32eccb-086b-4155-bb04-fe0a6d476daa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:43:50.363380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.363366    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b6f24bc-1abf-401a-a191-13435ba902b6-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:43:50.364972 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.364945    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a32eccb-086b-4155-bb04-fe0a6d476daa-kube-api-access-4ltw8" (OuterVolumeSpecName: "kube-api-access-4ltw8") pod "6a32eccb-086b-4155-bb04-fe0a6d476daa" (UID: "6a32eccb-086b-4155-bb04-fe0a6d476daa"). InnerVolumeSpecName "kube-api-access-4ltw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:50.365081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.365059    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6f24bc-1abf-401a-a191-13435ba902b6-kube-api-access-frjzc" (OuterVolumeSpecName: "kube-api-access-frjzc") pod "6b6f24bc-1abf-401a-a191-13435ba902b6" (UID: "6b6f24bc-1abf-401a-a191-13435ba902b6"). InnerVolumeSpecName "kube-api-access-frjzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:43:50.464808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.464766    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ltw8\" (UniqueName: \"kubernetes.io/projected/6a32eccb-086b-4155-bb04-fe0a6d476daa-kube-api-access-4ltw8\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:43:50.464808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.464800    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frjzc\" (UniqueName: \"kubernetes.io/projected/6b6f24bc-1abf-401a-a191-13435ba902b6-kube-api-access-frjzc\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:43:50.464808 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:50.464811    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a32eccb-086b-4155-bb04-fe0a6d476daa-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:43:51.192047 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.192018    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"
Sep 15 11:43:51.192532 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.192015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82" event={"ID":"6b6f24bc-1abf-401a-a191-13435ba902b6","Type":"ContainerDied","Data":"8cc83f2680bc613c8501437b2192006a4b0ca2b9d80305098ebc6b9cc1562b90"}
Sep 15 11:43:51.192532 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.192180    2574 scope.go:117] "RemoveContainer" containerID="69a2cace135bd6ccc0a0a4f5666a0fbd3d4ec34aed3befca82187fb3648bdaf6"
Sep 15 11:43:51.193618 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.193578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m" event={"ID":"6a32eccb-086b-4155-bb04-fe0a6d476daa","Type":"ContainerDied","Data":"1b81ec98a0e3fc761d6e3e9958738692a9a47af942e3a08dbf1d12d99620b757"}
Sep 15 11:43:51.193747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.193639    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"
Sep 15 11:43:51.201466 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.201449    2574 scope.go:117] "RemoveContainer" containerID="968cb58c6d88aae32110c8682be3819282087877bb7f290c599893de64f66229"
Sep 15 11:43:51.210178 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.210159    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"]
Sep 15 11:43:51.213914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.213893    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-53d3db96-6776578494-vdp82"]
Sep 15 11:43:51.223058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.223037    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"]
Sep 15 11:43:51.228283 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:51.228262    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-53d3db96-597dfdff85-x9f7m"]
Sep 15 11:43:52.666275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:52.666243    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a32eccb-086b-4155-bb04-fe0a6d476daa" path="/var/lib/kubelet/pods/6a32eccb-086b-4155-bb04-fe0a6d476daa/volumes"
Sep 15 11:43:52.666659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:43:52.666619    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6f24bc-1abf-401a-a191-13435ba902b6" path="/var/lib/kubelet/pods/6b6f24bc-1abf-401a-a191-13435ba902b6/volumes"
Sep 15 11:44:17.309032 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.308996    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"]
Sep 15 11:44:17.309531 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.309514    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6f24bc-1abf-401a-a191-13435ba902b6" containerName="payload-pre-processing"
Sep 15 11:44:17.309531 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.309532    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6f24bc-1abf-401a-a191-13435ba902b6" containerName="payload-pre-processing"
Sep 15 11:44:17.309649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.309549    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a32eccb-086b-4155-bb04-fe0a6d476daa" containerName="payload-processing"
Sep 15 11:44:17.309649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.309556    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a32eccb-086b-4155-bb04-fe0a6d476daa" containerName="payload-processing"
Sep 15 11:44:17.309649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.309619    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a32eccb-086b-4155-bb04-fe0a6d476daa" containerName="payload-processing"
Sep 15 11:44:17.309649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.309630    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6f24bc-1abf-401a-a191-13435ba902b6" containerName="payload-pre-processing"
Sep 15 11:44:17.313990 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.313972    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.316992 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.316970    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-01611d\""
Sep 15 11:44:17.317090 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.317064    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-01611d-dockercfg-rldth\""
Sep 15 11:44:17.325542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.325488    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"]
Sep 15 11:44:17.435863 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.435826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdf6l\" (UniqueName: \"kubernetes.io/projected/6bb6ad2b-2133-44f7-8874-528ba6354574-kube-api-access-gdf6l\") pod \"payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.436032 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.435895    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb6ad2b-2133-44f7-8874-528ba6354574-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.536693 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.536652    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb6ad2b-2133-44f7-8874-528ba6354574-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.536875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.536794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdf6l\" (UniqueName: \"kubernetes.io/projected/6bb6ad2b-2133-44f7-8874-528ba6354574-kube-api-access-gdf6l\") pod \"payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.537509 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.537475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb6ad2b-2133-44f7-8874-528ba6354574-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.545738 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.545718    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdf6l\" (UniqueName: \"kubernetes.io/projected/6bb6ad2b-2133-44f7-8874-528ba6354574-kube-api-access-gdf6l\") pod \"payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.625431 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.625347    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:17.896270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:17.896244    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"]
Sep 15 11:44:17.898759 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:44:17.898730    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb6ad2b_2133_44f7_8874_528ba6354574.slice/crio-5fc47ee42e539ffa0c23eb13b9af202d37d0b95f2137bd8f2cbe19fea71b7c6c WatchSource:0}: Error finding container 5fc47ee42e539ffa0c23eb13b9af202d37d0b95f2137bd8f2cbe19fea71b7c6c: Status 404 returned error can't find the container with id 5fc47ee42e539ffa0c23eb13b9af202d37d0b95f2137bd8f2cbe19fea71b7c6c
Sep 15 11:44:18.350849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:18.350760    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng" event={"ID":"6bb6ad2b-2133-44f7-8874-528ba6354574","Type":"ContainerStarted","Data":"82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753"}
Sep 15 11:44:18.350849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:18.350818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng" event={"ID":"6bb6ad2b-2133-44f7-8874-528ba6354574","Type":"ContainerStarted","Data":"5fc47ee42e539ffa0c23eb13b9af202d37d0b95f2137bd8f2cbe19fea71b7c6c"}
Sep 15 11:44:18.350849 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:18.350847    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:18.368914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:18.368866    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng" podStartSLOduration=1.368852476 podStartE2EDuration="1.368852476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:18.367330757 +0000 UTC m=+1152.145007015" watchObservedRunningTime="2026-09-15 11:44:18.368852476 +0000 UTC m=+1152.146528733"
Sep 15 11:44:21.757382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.757348    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"]
Sep 15 11:44:21.760906 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.760891    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.763038 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.763015    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"e2e-conflict-b-f81d2f5d-kserve-self-signed-certs\""
Sep 15 11:44:21.771649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.771627    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"]
Sep 15 11:44:21.883277 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.883240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlvw8\" (UniqueName: \"kubernetes.io/projected/4977d65d-a090-4325-9ac1-4c02ae939e47-kube-api-access-wlvw8\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.883461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.883295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977d65d-a090-4325-9ac1-4c02ae939e47-tls-certs\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.883461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.883390    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-model-cache\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.883461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.883443    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-tmp-dir\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.883616 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.883474    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-dshm\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.883616 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.883542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-home\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984098    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-tmp-dir\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984141    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-dshm\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984346 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-home\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984346 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984229    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlvw8\" (UniqueName: \"kubernetes.io/projected/4977d65d-a090-4325-9ac1-4c02ae939e47-kube-api-access-wlvw8\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984346 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977d65d-a090-4325-9ac1-4c02ae939e47-tls-certs\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984346 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-model-cache\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-tmp-dir\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984611    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-home\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.984811 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.984690    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-model-cache\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.986530 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.986511    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-dshm\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.986819 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.986797    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977d65d-a090-4325-9ac1-4c02ae939e47-tls-certs\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:21.993334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:21.993310    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlvw8\" (UniqueName: \"kubernetes.io/projected/4977d65d-a090-4325-9ac1-4c02ae939e47-kube-api-access-wlvw8\") pod \"e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") " pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:22.070323 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:22.070234    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:22.346171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:22.346141    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"]
Sep 15 11:44:22.347484 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:44:22.347456    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4977d65d_a090_4325_9ac1_4c02ae939e47.slice/crio-56979b2f17980c58f1c3ca83c3ae05a90ce143bc2c4b63344d1ff23f8f6c8e6f WatchSource:0}: Error finding container 56979b2f17980c58f1c3ca83c3ae05a90ce143bc2c4b63344d1ff23f8f6c8e6f: Status 404 returned error can't find the container with id 56979b2f17980c58f1c3ca83c3ae05a90ce143bc2c4b63344d1ff23f8f6c8e6f
Sep 15 11:44:22.372948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:22.372915    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx" event={"ID":"4977d65d-a090-4325-9ac1-4c02ae939e47","Type":"ContainerStarted","Data":"56979b2f17980c58f1c3ca83c3ae05a90ce143bc2c4b63344d1ff23f8f6c8e6f"}
Sep 15 11:44:23.380142 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:23.380107    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx" event={"ID":"4977d65d-a090-4325-9ac1-4c02ae939e47","Type":"ContainerStarted","Data":"6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c"}
Sep 15 11:44:23.417699 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:23.417652    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx" podStartSLOduration=2.41763566 podStartE2EDuration="2.41763566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:23.416400053 +0000 UTC m=+1157.194076322" watchObservedRunningTime="2026-09-15 11:44:23.41763566 +0000 UTC m=+1157.195311916"
Sep 15 11:44:29.357947 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:29.357864    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:44:32.070913 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:32.070882    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:32.071323 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:32.070932    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:32.083850 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:32.083819    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:32.440207 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:32.440179    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:44:33.137304 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.137268    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"]
Sep 15 11:44:33.137789 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.137490    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw" podUID="747f1974-0432-4f5c-ac0d-9ab12a0074cc" containerName="payload-processing" containerID="cri-o://0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895" gracePeriod=30
Sep 15 11:44:33.145024 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.144716    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"]
Sep 15 11:44:33.145024 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.144933    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5" podUID="ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" containerName="payload-pre-processing" containerID="cri-o://b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2" gracePeriod=30
Sep 15 11:44:33.430386 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.430359    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:44:33.433126 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.433101    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:44:33.435337 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.435309    2574 generic.go:356] "Generic (PLEG): container finished" podID="ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" containerID="b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2" exitCode=0
Sep 15 11:44:33.435463 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.435352    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"
Sep 15 11:44:33.435463 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.435392    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5" event={"ID":"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b","Type":"ContainerDied","Data":"b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2"}
Sep 15 11:44:33.435463 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.435435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5" event={"ID":"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b","Type":"ContainerDied","Data":"65dd160105c06b1e328c157d6bf2fe6c19da214309449ecc5ce215f967e00b21"}
Sep 15 11:44:33.435463 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.435462    2574 scope.go:117] "RemoveContainer" containerID="b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2"
Sep 15 11:44:33.436773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.436748    2574 generic.go:356] "Generic (PLEG): container finished" podID="747f1974-0432-4f5c-ac0d-9ab12a0074cc" containerID="0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895" exitCode=0
Sep 15 11:44:33.436865 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.436784    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"
Sep 15 11:44:33.436865 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.436835    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw" event={"ID":"747f1974-0432-4f5c-ac0d-9ab12a0074cc","Type":"ContainerDied","Data":"0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895"}
Sep 15 11:44:33.436976 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.436880    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw" event={"ID":"747f1974-0432-4f5c-ac0d-9ab12a0074cc","Type":"ContainerDied","Data":"3821621ad5ba69c68d5c10fc0aefa7588c790b7bc05954b0ebdfea70209abaec"}
Sep 15 11:44:33.443920 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.443894    2574 scope.go:117] "RemoveContainer" containerID="b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2"
Sep 15 11:44:33.444183 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:44:33.444166    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2\": container with ID starting with b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2 not found: ID does not exist" containerID="b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2"
Sep 15 11:44:33.444228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.444190    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2"} err="failed to get container status \"b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2\": rpc error: code = NotFound desc = could not find container \"b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2\": container with ID starting with b3fd2fbc49eb13af8c35a0e1b26b1653df34aa8dfd6961401711d798ba2463b2 not found: ID does not exist"
Sep 15 11:44:33.444228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.444205    2574 scope.go:117] "RemoveContainer" containerID="0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895"
Sep 15 11:44:33.451864 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.451848    2574 scope.go:117] "RemoveContainer" containerID="0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895"
Sep 15 11:44:33.452156 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:44:33.452139    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895\": container with ID starting with 0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895 not found: ID does not exist" containerID="0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895"
Sep 15 11:44:33.452208 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.452162    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895"} err="failed to get container status \"0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895\": rpc error: code = NotFound desc = could not find container \"0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895\": container with ID starting with 0601659e99566e1f8c08e75af4da999468530b2ac2930ec1c5b68a7e1736e895 not found: ID does not exist"
Sep 15 11:44:33.495442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.495401    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-plugins-config-volume\") pod \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") "
Sep 15 11:44:33.495653 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.495462    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747f1974-0432-4f5c-ac0d-9ab12a0074cc-plugins-config-volume\") pod \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") "
Sep 15 11:44:33.495741 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.495660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk5g4\" (UniqueName: \"kubernetes.io/projected/747f1974-0432-4f5c-ac0d-9ab12a0074cc-kube-api-access-xk5g4\") pod \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\" (UID: \"747f1974-0432-4f5c-ac0d-9ab12a0074cc\") "
Sep 15 11:44:33.495797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.495753    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78j67\" (UniqueName: \"kubernetes.io/projected/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-kube-api-access-78j67\") pod \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\" (UID: \"ea3c4aea-a301-496c-8eae-dc59ed5dfe1b\") "
Sep 15 11:44:33.495963 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.495930    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" (UID: "ea3c4aea-a301-496c-8eae-dc59ed5dfe1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:44:33.496046 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.496022    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747f1974-0432-4f5c-ac0d-9ab12a0074cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "747f1974-0432-4f5c-ac0d-9ab12a0074cc" (UID: "747f1974-0432-4f5c-ac0d-9ab12a0074cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:44:33.496474 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.496454    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:44:33.496474 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.496473    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/747f1974-0432-4f5c-ac0d-9ab12a0074cc-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:44:33.498450 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.498420    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-kube-api-access-78j67" (OuterVolumeSpecName: "kube-api-access-78j67") pod "ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" (UID: "ea3c4aea-a301-496c-8eae-dc59ed5dfe1b"). InnerVolumeSpecName "kube-api-access-78j67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:44:33.498571 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.498546    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747f1974-0432-4f5c-ac0d-9ab12a0074cc-kube-api-access-xk5g4" (OuterVolumeSpecName: "kube-api-access-xk5g4") pod "747f1974-0432-4f5c-ac0d-9ab12a0074cc" (UID: "747f1974-0432-4f5c-ac0d-9ab12a0074cc"). InnerVolumeSpecName "kube-api-access-xk5g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:44:33.597832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.597798    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5g4\" (UniqueName: \"kubernetes.io/projected/747f1974-0432-4f5c-ac0d-9ab12a0074cc-kube-api-access-xk5g4\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:44:33.597832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.597829    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78j67\" (UniqueName: \"kubernetes.io/projected/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b-kube-api-access-78j67\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:44:33.759013 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.758983    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"]
Sep 15 11:44:33.762672 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.762651    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0eb0ef-86b49b8d6-bpsl5"]
Sep 15 11:44:33.773265 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.773242    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"]
Sep 15 11:44:33.777076 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:33.777058    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0eb0ef-55f6bf4d7c-t26fw"]
Sep 15 11:44:34.671903 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:34.671863    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="747f1974-0432-4f5c-ac0d-9ab12a0074cc" path="/var/lib/kubelet/pods/747f1974-0432-4f5c-ac0d-9ab12a0074cc/volumes"
Sep 15 11:44:34.672342 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:34.672325    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" path="/var/lib/kubelet/pods/ea3c4aea-a301-496c-8eae-dc59ed5dfe1b/volumes"
Sep 15 11:44:35.864837 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.864802    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"]
Sep 15 11:44:35.865235 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.865223    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" containerName="payload-pre-processing"
Sep 15 11:44:35.865235 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.865236    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" containerName="payload-pre-processing"
Sep 15 11:44:35.865310 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.865268    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="747f1974-0432-4f5c-ac0d-9ab12a0074cc" containerName="payload-processing"
Sep 15 11:44:35.865310 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.865273    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="747f1974-0432-4f5c-ac0d-9ab12a0074cc" containerName="payload-processing"
Sep 15 11:44:35.865378 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.865334    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="747f1974-0432-4f5c-ac0d-9ab12a0074cc" containerName="payload-processing"
Sep 15 11:44:35.865378 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.865344    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea3c4aea-a301-496c-8eae-dc59ed5dfe1b" containerName="payload-pre-processing"
Sep 15 11:44:35.868375 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.868359    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:35.870305 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.870280    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4dbbbf-dockercfg-bmgxs\""
Sep 15 11:44:35.870437 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.870348    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4dbbbf\""
Sep 15 11:44:35.876891 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.876868    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"]
Sep 15 11:44:35.922055 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.922024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713dcd-c36c-4899-a060-907aa14dffad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:35.922055 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:35.922062    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ed713dcd-c36c-4899-a060-907aa14dffad-kube-api-access-qngm7\") pod \"payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:36.023426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:36.023388    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713dcd-c36c-4899-a060-907aa14dffad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:36.023426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:36.023429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ed713dcd-c36c-4899-a060-907aa14dffad-kube-api-access-qngm7\") pod \"payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:36.024220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:36.024198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713dcd-c36c-4899-a060-907aa14dffad-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:36.032845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:36.032809    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ed713dcd-c36c-4899-a060-907aa14dffad-kube-api-access-qngm7\") pod \"payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:36.180037 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:36.179999    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:36.450616 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:36.450587    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"]
Sep 15 11:44:36.452132 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:44:36.452105    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded713dcd_c36c_4899_a060_907aa14dffad.slice/crio-f0bbb7098e33b68566c36e2abbe9a9ac1892614e97d7e8ccf738f8825fc92a9d WatchSource:0}: Error finding container f0bbb7098e33b68566c36e2abbe9a9ac1892614e97d7e8ccf738f8825fc92a9d: Status 404 returned error can't find the container with id f0bbb7098e33b68566c36e2abbe9a9ac1892614e97d7e8ccf738f8825fc92a9d
Sep 15 11:44:37.464407 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:37.464361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb" event={"ID":"ed713dcd-c36c-4899-a060-907aa14dffad","Type":"ContainerStarted","Data":"594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794"}
Sep 15 11:44:37.464803 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:37.464425    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb" event={"ID":"ed713dcd-c36c-4899-a060-907aa14dffad","Type":"ContainerStarted","Data":"f0bbb7098e33b68566c36e2abbe9a9ac1892614e97d7e8ccf738f8825fc92a9d"}
Sep 15 11:44:37.464803 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:37.464479    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:44:37.480079 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:37.479989    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb" podStartSLOduration=2.47996613 podStartE2EDuration="2.47996613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:44:37.479825682 +0000 UTC m=+1171.257501941" watchObservedRunningTime="2026-09-15 11:44:37.47996613 +0000 UTC m=+1171.257642368"
Sep 15 11:44:48.472344 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:44:48.472312    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:45:00.157388 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.157353    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"]
Sep 15 11:45:00.162180 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.162159    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"]
Sep 15 11:45:00.162334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.162293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"
Sep 15 11:45:00.165535 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.165510    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:45:00.165872 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.165855    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"
Sep 15 11:45:00.172999 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.172983    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"
Sep 15 11:45:00.179720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.179697    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"]
Sep 15 11:45:00.182665 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.182645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"
Sep 15 11:45:00.182916 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.182900    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"]
Sep 15 11:45:00.185631 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.185608    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"]
Sep 15 11:45:00.191337 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.191298    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"
Sep 15 11:45:00.203520 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.203383    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"]
Sep 15 11:45:00.234015 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.233992    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"
Sep 15 11:45:00.462843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.462816    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"]
Sep 15 11:45:00.464422 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:45:00.464390    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9e9e9e_6d33_4f42_b7e8_70a80d71aaca.slice/crio-4ba59aad6809abcb4292b65ad2229141d8980d23ed3f4b109cc8c1a540d36d2b WatchSource:0}: Error finding container 4ba59aad6809abcb4292b65ad2229141d8980d23ed3f4b109cc8c1a540d36d2b: Status 404 returned error can't find the container with id 4ba59aad6809abcb4292b65ad2229141d8980d23ed3f4b109cc8c1a540d36d2b
Sep 15 11:45:00.466221 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.466206    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 11:45:00.486776 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.486723    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"]
Sep 15 11:45:00.488890 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:45:00.488862    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711f052c_4454_4964_8c82_d5bc06c6b839.slice/crio-b50fa18ea43eef02e4ea09fb0eede6a18c982817753e986c6d2f1d5bcd730fbc WatchSource:0}: Error finding container b50fa18ea43eef02e4ea09fb0eede6a18c982817753e986c6d2f1d5bcd730fbc: Status 404 returned error can't find the container with id b50fa18ea43eef02e4ea09fb0eede6a18c982817753e986c6d2f1d5bcd730fbc
Sep 15 11:45:00.528843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.528818    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"]
Sep 15 11:45:00.530033 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:45:00.530010    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b497d16_5273_46cc_a072_03a412e422ff.slice/crio-da819bdcb69298627c7131eeda898fcdfb24cf5e2ca166b3972971a46adcc559 WatchSource:0}: Error finding container da819bdcb69298627c7131eeda898fcdfb24cf5e2ca166b3972971a46adcc559: Status 404 returned error can't find the container with id da819bdcb69298627c7131eeda898fcdfb24cf5e2ca166b3972971a46adcc559
Sep 15 11:45:00.599159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.599120    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6" event={"ID":"711f052c-4454-4964-8c82-d5bc06c6b839","Type":"ContainerStarted","Data":"b50fa18ea43eef02e4ea09fb0eede6a18c982817753e986c6d2f1d5bcd730fbc"}
Sep 15 11:45:00.600368 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.600336    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2" event={"ID":"9b497d16-5273-46cc-a072-03a412e422ff","Type":"ContainerStarted","Data":"da819bdcb69298627c7131eeda898fcdfb24cf5e2ca166b3972971a46adcc559"}
Sep 15 11:45:00.601542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:00.601516    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5" event={"ID":"7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca","Type":"ContainerStarted","Data":"4ba59aad6809abcb4292b65ad2229141d8980d23ed3f4b109cc8c1a540d36d2b"}
Sep 15 11:45:03.624897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.624801    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6" event={"ID":"711f052c-4454-4964-8c82-d5bc06c6b839","Type":"ContainerStarted","Data":"4370617d23185c4c14825edc0552e5942dd07c0b4c5d14c0bbf5843917afbfc0"}
Sep 15 11:45:03.626377 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.626343    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2" event={"ID":"9b497d16-5273-46cc-a072-03a412e422ff","Type":"ContainerStarted","Data":"651bb1e66030d8283add380f33283b802cc83b4fae7738b0e493512e2969507b"}
Sep 15 11:45:03.627809 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.627781    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5" event={"ID":"7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca","Type":"ContainerStarted","Data":"a73bbae3208ee2e4a4a08c9c789ec200b1e1e68ed145bb9a99276a6facbf1cb3"}
Sep 15 11:45:03.640449 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.640396    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6" podStartSLOduration=1.0988785 podStartE2EDuration="3.640377008s" totalImagesPullingTime="2.541498508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:03.639525169 +0000 UTC m=+1197.417201417" watchObservedRunningTime="2026-09-15 11:45:03.640377008 +0000 UTC m=+1197.418053263"
Sep 15 11:45:03.654232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.654155    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2" podStartSLOduration=1.157719913 podStartE2EDuration="3.654137458s" totalImagesPullingTime="2.496417545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:03.653453658 +0000 UTC m=+1197.431129909" watchObservedRunningTime="2026-09-15 11:45:03.654137458 +0000 UTC m=+1197.431813714"
Sep 15 11:45:03.672226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.672174    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5" podStartSLOduration=1.110016639 podStartE2EDuration="3.672158663s" totalImagesPullingTime="2.562142024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:03.670034809 +0000 UTC m=+1197.447711067" watchObservedRunningTime="2026-09-15 11:45:03.672158663 +0000 UTC m=+1197.449834920"
Sep 15 11:45:03.969068 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.969037    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"]
Sep 15 11:45:03.972763 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.972743    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:03.975164 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.975139    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-849a35\"/\"openshift-service-ca.crt\""
Sep 15 11:45:03.975749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.975730    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-849a35\"/\"noten-model-849a35-kserve-self-signed-certs\""
Sep 15 11:45:03.975843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.975763    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-849a35\"/\"kube-root-ca.crt\""
Sep 15 11:45:03.975843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.975731    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-849a35\"/\"default-dockercfg-qk6jt\""
Sep 15 11:45:03.981908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:03.981886    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"]
Sep 15 11:45:04.007826 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.007797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f8h\" (UniqueName: \"kubernetes.io/projected/1e4bde6d-662c-4c21-b82b-bd15902618b6-kube-api-access-p6f8h\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.007996 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.007865    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-tmp-dir\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.007996 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.007929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-home\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.007996 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.007958    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4bde6d-662c-4c21-b82b-bd15902618b6-tls-certs\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.008129 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.008074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-dshm\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.008129 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.008101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-model-cache\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.108670 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.108640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-dshm\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.108795 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.108679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-model-cache\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.108795 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.108711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f8h\" (UniqueName: \"kubernetes.io/projected/1e4bde6d-662c-4c21-b82b-bd15902618b6-kube-api-access-p6f8h\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.108795 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.108756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-tmp-dir\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.108795 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.108777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-home\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.108973 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.108800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4bde6d-662c-4c21-b82b-bd15902618b6-tls-certs\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.109138 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.109117    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-model-cache\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.109228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.109168    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-tmp-dir\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.109228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.109198    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-home\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.110979 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.110947    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-dshm\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.111371 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.111352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4bde6d-662c-4c21-b82b-bd15902618b6-tls-certs\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.116479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.116462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f8h\" (UniqueName: \"kubernetes.io/projected/1e4bde6d-662c-4c21-b82b-bd15902618b6-kube-api-access-p6f8h\") pod \"noten-model-849a35-kserve-7fc486f495-9qjxh\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") " pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.282982 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.282900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:04.556788 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.556761    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"]
Sep 15 11:45:04.558907 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:45:04.558879    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e4bde6d_662c_4c21_b82b_bd15902618b6.slice/crio-6fe1ba119f6ece917eac4783558f4aaf64876c2f31dfebcb23a0a34dafc3c4c3 WatchSource:0}: Error finding container 6fe1ba119f6ece917eac4783558f4aaf64876c2f31dfebcb23a0a34dafc3c4c3: Status 404 returned error can't find the container with id 6fe1ba119f6ece917eac4783558f4aaf64876c2f31dfebcb23a0a34dafc3c4c3
Sep 15 11:45:04.636439 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.636407    2574 generic.go:356] "Generic (PLEG): container finished" podID="9b497d16-5273-46cc-a072-03a412e422ff" containerID="651bb1e66030d8283add380f33283b802cc83b4fae7738b0e493512e2969507b" exitCode=0
Sep 15 11:45:04.636931 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.636448    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2" event={"ID":"9b497d16-5273-46cc-a072-03a412e422ff","Type":"ContainerDied","Data":"651bb1e66030d8283add380f33283b802cc83b4fae7738b0e493512e2969507b"}
Sep 15 11:45:04.638356 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.638329    2574 generic.go:356] "Generic (PLEG): container finished" podID="7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca" containerID="a73bbae3208ee2e4a4a08c9c789ec200b1e1e68ed145bb9a99276a6facbf1cb3" exitCode=0
Sep 15 11:45:04.638490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.638394    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5" event={"ID":"7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca","Type":"ContainerDied","Data":"a73bbae3208ee2e4a4a08c9c789ec200b1e1e68ed145bb9a99276a6facbf1cb3"}
Sep 15 11:45:04.640303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.640276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh" event={"ID":"1e4bde6d-662c-4c21-b82b-bd15902618b6","Type":"ContainerStarted","Data":"6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c"}
Sep 15 11:45:04.640419 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.640313    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh" event={"ID":"1e4bde6d-662c-4c21-b82b-bd15902618b6","Type":"ContainerStarted","Data":"6fe1ba119f6ece917eac4783558f4aaf64876c2f31dfebcb23a0a34dafc3c4c3"}
Sep 15 11:45:04.642208 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.642184    2574 generic.go:356] "Generic (PLEG): container finished" podID="711f052c-4454-4964-8c82-d5bc06c6b839" containerID="4370617d23185c4c14825edc0552e5942dd07c0b4c5d14c0bbf5843917afbfc0" exitCode=0
Sep 15 11:45:04.642290 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.642254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6" event={"ID":"711f052c-4454-4964-8c82-d5bc06c6b839","Type":"ContainerDied","Data":"4370617d23185c4c14825edc0552e5942dd07c0b4c5d14c0bbf5843917afbfc0"}
Sep 15 11:45:04.669675 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:04.669627    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh" podStartSLOduration=1.669610998 podStartE2EDuration="1.669610998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:04.66898246 +0000 UTC m=+1198.446658717" watchObservedRunningTime="2026-09-15 11:45:04.669610998 +0000 UTC m=+1198.447287255"
Sep 15 11:45:05.842674 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:05.842643    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"
Sep 15 11:45:05.865855 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:05.865832    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"
Sep 15 11:45:05.873127 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:05.873100    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"
Sep 15 11:45:06.654644 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.654608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6" event={"ID":"711f052c-4454-4964-8c82-d5bc06c6b839","Type":"ContainerDied","Data":"b50fa18ea43eef02e4ea09fb0eede6a18c982817753e986c6d2f1d5bcd730fbc"}
Sep 15 11:45:06.654856 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.654659    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b50fa18ea43eef02e4ea09fb0eede6a18c982817753e986c6d2f1d5bcd730fbc"
Sep 15 11:45:06.654856 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.654714    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"
Sep 15 11:45:06.656624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.656118    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"
Sep 15 11:45:06.656624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.656123    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2" event={"ID":"9b497d16-5273-46cc-a072-03a412e422ff","Type":"ContainerDied","Data":"da819bdcb69298627c7131eeda898fcdfb24cf5e2ca166b3972971a46adcc559"}
Sep 15 11:45:06.656624 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.656168    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da819bdcb69298627c7131eeda898fcdfb24cf5e2ca166b3972971a46adcc559"
Sep 15 11:45:06.658193 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.657753    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5"
Sep 15 11:45:06.658193 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.657773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29824545-867x5" event={"ID":"7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca","Type":"ContainerDied","Data":"4ba59aad6809abcb4292b65ad2229141d8980d23ed3f4b109cc8c1a540d36d2b"}
Sep 15 11:45:06.658193 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.657813    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba59aad6809abcb4292b65ad2229141d8980d23ed3f4b109cc8c1a540d36d2b"
Sep 15 11:45:06.689159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.689134    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:45:06.692267 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.692243    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:45:06.693560 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.693538    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:45:06.696629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:06.696613    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:45:14.283332 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:14.283295    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:14.283731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:14.283353    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:14.295985 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:14.295948    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:14.710326 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:14.710295    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:19.547835 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.547798    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"]
Sep 15 11:45:19.548274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548260    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca" containerName="cleanup"
Sep 15 11:45:19.548313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548275    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca" containerName="cleanup"
Sep 15 11:45:19.548313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548296    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b497d16-5273-46cc-a072-03a412e422ff" containerName="cleanup"
Sep 15 11:45:19.548313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548301    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b497d16-5273-46cc-a072-03a412e422ff" containerName="cleanup"
Sep 15 11:45:19.548313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548312    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711f052c-4454-4964-8c82-d5bc06c6b839" containerName="cleanup"
Sep 15 11:45:19.548452 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548318    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="711f052c-4454-4964-8c82-d5bc06c6b839" containerName="cleanup"
Sep 15 11:45:19.548452 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548381    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca" containerName="cleanup"
Sep 15 11:45:19.548452 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548388    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b497d16-5273-46cc-a072-03a412e422ff" containerName="cleanup"
Sep 15 11:45:19.548452 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.548395    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="711f052c-4454-4964-8c82-d5bc06c6b839" containerName="cleanup"
Sep 15 11:45:19.552705 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.552689    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.555002 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.554944    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-944c275f\""
Sep 15 11:45:19.555138 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.555045    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-944c275f-dockercfg-kjcm7\""
Sep 15 11:45:19.561980 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.561953    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"]
Sep 15 11:45:19.579092 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.579063    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"]
Sep 15 11:45:19.583620 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.583600    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.592149 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.592124    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"]
Sep 15 11:45:19.659971 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.659932    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4dkn\" (UniqueName: \"kubernetes.io/projected/3a87897e-09e3-4969-bfd1-0d2f52080641-kube-api-access-s4dkn\") pod \"payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") " pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.660157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.659976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkrx7\" (UniqueName: \"kubernetes.io/projected/7cf34740-a9cf-4d58-b8c6-bae94076ba66-kube-api-access-tkrx7\") pod \"payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.660157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.660072    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf34740-a9cf-4d58-b8c6-bae94076ba66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.660157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.660130    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a87897e-09e3-4969-bfd1-0d2f52080641-plugins-config-volume\") pod \"payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") " pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.760765 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.760730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf34740-a9cf-4d58-b8c6-bae94076ba66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.760964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.760799    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a87897e-09e3-4969-bfd1-0d2f52080641-plugins-config-volume\") pod \"payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") " pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.760964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.760917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4dkn\" (UniqueName: \"kubernetes.io/projected/3a87897e-09e3-4969-bfd1-0d2f52080641-kube-api-access-s4dkn\") pod \"payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") " pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.760964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.760951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkrx7\" (UniqueName: \"kubernetes.io/projected/7cf34740-a9cf-4d58-b8c6-bae94076ba66-kube-api-access-tkrx7\") pod \"payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.761562 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.761536    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf34740-a9cf-4d58-b8c6-bae94076ba66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.761648 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.761602    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a87897e-09e3-4969-bfd1-0d2f52080641-plugins-config-volume\") pod \"payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") " pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.768607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.768583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4dkn\" (UniqueName: \"kubernetes.io/projected/3a87897e-09e3-4969-bfd1-0d2f52080641-kube-api-access-s4dkn\") pod \"payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") " pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:19.768834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.768814    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkrx7\" (UniqueName: \"kubernetes.io/projected/7cf34740-a9cf-4d58-b8c6-bae94076ba66-kube-api-access-tkrx7\") pod \"payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.864176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.864094    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:19.894811 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:19.894781    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:20.137829 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.137800    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"]
Sep 15 11:45:20.139011 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:45:20.138987    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf34740_a9cf_4d58_b8c6_bae94076ba66.slice/crio-066964c388c481323f3013b04d05c67c2ca094d7d01c1924707ab40b62f374b2 WatchSource:0}: Error finding container 066964c388c481323f3013b04d05c67c2ca094d7d01c1924707ab40b62f374b2: Status 404 returned error can't find the container with id 066964c388c481323f3013b04d05c67c2ca094d7d01c1924707ab40b62f374b2
Sep 15 11:45:20.177928 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.177898    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"]
Sep 15 11:45:20.183755 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:45:20.183714    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a87897e_09e3_4969_bfd1_0d2f52080641.slice/crio-20486d6fe67f5cf16c5a3c7f9e95c83e0120903a7157114bd5e7b194ee889467 WatchSource:0}: Error finding container 20486d6fe67f5cf16c5a3c7f9e95c83e0120903a7157114bd5e7b194ee889467: Status 404 returned error can't find the container with id 20486d6fe67f5cf16c5a3c7f9e95c83e0120903a7157114bd5e7b194ee889467
Sep 15 11:45:20.734003 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.733948    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k" event={"ID":"7cf34740-a9cf-4d58-b8c6-bae94076ba66","Type":"ContainerStarted","Data":"f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1"}
Sep 15 11:45:20.734489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.734010    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k" event={"ID":"7cf34740-a9cf-4d58-b8c6-bae94076ba66","Type":"ContainerStarted","Data":"066964c388c481323f3013b04d05c67c2ca094d7d01c1924707ab40b62f374b2"}
Sep 15 11:45:20.734489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.734113    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:20.735790 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.735761    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7" event={"ID":"3a87897e-09e3-4969-bfd1-0d2f52080641","Type":"ContainerStarted","Data":"bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f"}
Sep 15 11:45:20.735899 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.735804    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7" event={"ID":"3a87897e-09e3-4969-bfd1-0d2f52080641","Type":"ContainerStarted","Data":"20486d6fe67f5cf16c5a3c7f9e95c83e0120903a7157114bd5e7b194ee889467"}
Sep 15 11:45:20.735899 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.735882    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:20.750888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.750842    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k" podStartSLOduration=1.750830262 podStartE2EDuration="1.750830262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:20.749353337 +0000 UTC m=+1214.527029597" watchObservedRunningTime="2026-09-15 11:45:20.750830262 +0000 UTC m=+1214.528506519"
Sep 15 11:45:20.771750 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:20.771701    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7" podStartSLOduration=1.771684458 podStartE2EDuration="1.771684458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:45:20.768152092 +0000 UTC m=+1214.545828349" watchObservedRunningTime="2026-09-15 11:45:20.771684458 +0000 UTC m=+1214.549360716"
Sep 15 11:45:27.600125 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.600090    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"]
Sep 15 11:45:27.602212 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.602190    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1358dc-29824545-6g7t6"]
Sep 15 11:45:27.624423 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.624390    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"]
Sep 15 11:45:27.624921 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.624878    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt" podUID="626d9add-f766-4b85-bea5-8455f4b2882e" containerName="payload-processing" containerID="cri-o://c60a7f9caa1f06446d3991f95c7d2064c8cba49294041475f642ec14ee47df84" gracePeriod=30
Sep 15 11:45:27.636444 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.636416    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"]
Sep 15 11:45:27.636662 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.636636    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh" podUID="58aee0ab-904e-4ff0-8f17-a311d714b22f" containerName="payload-pre-processing" containerID="cri-o://a7ef2c7856f318c9e3bdad2e3174ce29b6980fa2b2ae54924d53a880f673bf09" gracePeriod=30
Sep 15 11:45:27.786636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.786604    2574 generic.go:356] "Generic (PLEG): container finished" podID="626d9add-f766-4b85-bea5-8455f4b2882e" containerID="c60a7f9caa1f06446d3991f95c7d2064c8cba49294041475f642ec14ee47df84" exitCode=0
Sep 15 11:45:27.786793 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.786666    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt" event={"ID":"626d9add-f766-4b85-bea5-8455f4b2882e","Type":"ContainerDied","Data":"c60a7f9caa1f06446d3991f95c7d2064c8cba49294041475f642ec14ee47df84"}
Sep 15 11:45:27.788562 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.788538    2574 generic.go:356] "Generic (PLEG): container finished" podID="58aee0ab-904e-4ff0-8f17-a311d714b22f" containerID="a7ef2c7856f318c9e3bdad2e3174ce29b6980fa2b2ae54924d53a880f673bf09" exitCode=0
Sep 15 11:45:27.788682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.788598    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh" event={"ID":"58aee0ab-904e-4ff0-8f17-a311d714b22f","Type":"ContainerDied","Data":"a7ef2c7856f318c9e3bdad2e3174ce29b6980fa2b2ae54924d53a880f673bf09"}
Sep 15 11:45:27.915069 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.915050    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:45:27.918233 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:27.918215    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:45:28.042331 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042295    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jft4h\" (UniqueName: \"kubernetes.io/projected/58aee0ab-904e-4ff0-8f17-a311d714b22f-kube-api-access-jft4h\") pod \"58aee0ab-904e-4ff0-8f17-a311d714b22f\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") "
Sep 15 11:45:28.042491 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042376    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aee0ab-904e-4ff0-8f17-a311d714b22f-plugins-config-volume\") pod \"58aee0ab-904e-4ff0-8f17-a311d714b22f\" (UID: \"58aee0ab-904e-4ff0-8f17-a311d714b22f\") "
Sep 15 11:45:28.042491 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042442    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626d9add-f766-4b85-bea5-8455f4b2882e-plugins-config-volume\") pod \"626d9add-f766-4b85-bea5-8455f4b2882e\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") "
Sep 15 11:45:28.042628 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042564    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7brt7\" (UniqueName: \"kubernetes.io/projected/626d9add-f766-4b85-bea5-8455f4b2882e-kube-api-access-7brt7\") pod \"626d9add-f766-4b85-bea5-8455f4b2882e\" (UID: \"626d9add-f766-4b85-bea5-8455f4b2882e\") "
Sep 15 11:45:28.042830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042806    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/626d9add-f766-4b85-bea5-8455f4b2882e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "626d9add-f766-4b85-bea5-8455f4b2882e" (UID: "626d9add-f766-4b85-bea5-8455f4b2882e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:45:28.042909 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042804    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aee0ab-904e-4ff0-8f17-a311d714b22f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58aee0ab-904e-4ff0-8f17-a311d714b22f" (UID: "58aee0ab-904e-4ff0-8f17-a311d714b22f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:45:28.042909 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042901    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/626d9add-f766-4b85-bea5-8455f4b2882e-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:28.043020 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.042915    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58aee0ab-904e-4ff0-8f17-a311d714b22f-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:28.044561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.044524    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/626d9add-f766-4b85-bea5-8455f4b2882e-kube-api-access-7brt7" (OuterVolumeSpecName: "kube-api-access-7brt7") pod "626d9add-f766-4b85-bea5-8455f4b2882e" (UID: "626d9add-f766-4b85-bea5-8455f4b2882e"). InnerVolumeSpecName "kube-api-access-7brt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:45:28.044561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.044540    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aee0ab-904e-4ff0-8f17-a311d714b22f-kube-api-access-jft4h" (OuterVolumeSpecName: "kube-api-access-jft4h") pod "58aee0ab-904e-4ff0-8f17-a311d714b22f" (UID: "58aee0ab-904e-4ff0-8f17-a311d714b22f"). InnerVolumeSpecName "kube-api-access-jft4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:45:28.143540 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.143482    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7brt7\" (UniqueName: \"kubernetes.io/projected/626d9add-f766-4b85-bea5-8455f4b2882e-kube-api-access-7brt7\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:28.143540 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.143540    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jft4h\" (UniqueName: \"kubernetes.io/projected/58aee0ab-904e-4ff0-8f17-a311d714b22f-kube-api-access-jft4h\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:28.666876 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.666844    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="711f052c-4454-4964-8c82-d5bc06c6b839" path="/var/lib/kubelet/pods/711f052c-4454-4964-8c82-d5bc06c6b839/volumes"
Sep 15 11:45:28.796033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.795996    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"
Sep 15 11:45:28.796033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.796011    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh" event={"ID":"58aee0ab-904e-4ff0-8f17-a311d714b22f","Type":"ContainerDied","Data":"f8fafd24f978e83c1e57d2ca76f32b216fc1b0cf6dc0d60e5863ad5d9222f001"}
Sep 15 11:45:28.796298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.796108    2574 scope.go:117] "RemoveContainer" containerID="a7ef2c7856f318c9e3bdad2e3174ce29b6980fa2b2ae54924d53a880f673bf09"
Sep 15 11:45:28.798009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.797989    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"
Sep 15 11:45:28.798091 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.798013    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt" event={"ID":"626d9add-f766-4b85-bea5-8455f4b2882e","Type":"ContainerDied","Data":"75ae9af985daff1d7496eb72f5d326c4de973c7c890474097084aa83b045fc79"}
Sep 15 11:45:28.805700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.805684    2574 scope.go:117] "RemoveContainer" containerID="c60a7f9caa1f06446d3991f95c7d2064c8cba49294041475f642ec14ee47df84"
Sep 15 11:45:28.816462 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.816441    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"]
Sep 15 11:45:28.820464 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.820442    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-1358dc-7c587f6458-p5knh"]
Sep 15 11:45:28.830182 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.830153    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"]
Sep 15 11:45:28.835232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:28.835210    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1358dc-7cdc6bdc5f-nn2lt"]
Sep 15 11:45:30.666014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:30.665982    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58aee0ab-904e-4ff0-8f17-a311d714b22f" path="/var/lib/kubelet/pods/58aee0ab-904e-4ff0-8f17-a311d714b22f/volumes"
Sep 15 11:45:30.666699 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:30.666673    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="626d9add-f766-4b85-bea5-8455f4b2882e" path="/var/lib/kubelet/pods/626d9add-f766-4b85-bea5-8455f4b2882e/volumes"
Sep 15 11:45:31.745641 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:31.745611    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:45:31.746075 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:31.745683    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:45:44.051484 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:44.051446    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"]
Sep 15 11:45:59.070791 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.070732    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh" podUID="1e4bde6d-662c-4c21-b82b-bd15902618b6" containerName="main" containerID="cri-o://6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c" gracePeriod=45
Sep 15 11:45:59.326804 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.326746    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:59.455006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.454976    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-home\") pod \"1e4bde6d-662c-4c21-b82b-bd15902618b6\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") "
Sep 15 11:45:59.455188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455014    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6f8h\" (UniqueName: \"kubernetes.io/projected/1e4bde6d-662c-4c21-b82b-bd15902618b6-kube-api-access-p6f8h\") pod \"1e4bde6d-662c-4c21-b82b-bd15902618b6\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") "
Sep 15 11:45:59.455188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455064    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4bde6d-662c-4c21-b82b-bd15902618b6-tls-certs\") pod \"1e4bde6d-662c-4c21-b82b-bd15902618b6\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") "
Sep 15 11:45:59.455188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455084    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-model-cache\") pod \"1e4bde6d-662c-4c21-b82b-bd15902618b6\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") "
Sep 15 11:45:59.455188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455142    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-dshm\") pod \"1e4bde6d-662c-4c21-b82b-bd15902618b6\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") "
Sep 15 11:45:59.455402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455231    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-tmp-dir\") pod \"1e4bde6d-662c-4c21-b82b-bd15902618b6\" (UID: \"1e4bde6d-662c-4c21-b82b-bd15902618b6\") "
Sep 15 11:45:59.455402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455243    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-home" (OuterVolumeSpecName: "home") pod "1e4bde6d-662c-4c21-b82b-bd15902618b6" (UID: "1e4bde6d-662c-4c21-b82b-bd15902618b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:59.455402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455369    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e4bde6d-662c-4c21-b82b-bd15902618b6" (UID: "1e4bde6d-662c-4c21-b82b-bd15902618b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:59.455607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455492    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e4bde6d-662c-4c21-b82b-bd15902618b6" (UID: "1e4bde6d-662c-4c21-b82b-bd15902618b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:59.455607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455583    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:59.455607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.455604    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:59.457373 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.457336    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4bde6d-662c-4c21-b82b-bd15902618b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e4bde6d-662c-4c21-b82b-bd15902618b6" (UID: "1e4bde6d-662c-4c21-b82b-bd15902618b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:45:59.457474 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.457401    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4bde6d-662c-4c21-b82b-bd15902618b6-kube-api-access-p6f8h" (OuterVolumeSpecName: "kube-api-access-p6f8h") pod "1e4bde6d-662c-4c21-b82b-bd15902618b6" (UID: "1e4bde6d-662c-4c21-b82b-bd15902618b6"). InnerVolumeSpecName "kube-api-access-p6f8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:45:59.457474 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.457430    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-dshm" (OuterVolumeSpecName: "dshm") pod "1e4bde6d-662c-4c21-b82b-bd15902618b6" (UID: "1e4bde6d-662c-4c21-b82b-bd15902618b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:45:59.556708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.556679    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:59.556708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.556706    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6f8h\" (UniqueName: \"kubernetes.io/projected/1e4bde6d-662c-4c21-b82b-bd15902618b6-kube-api-access-p6f8h\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:59.556897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.556716    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e4bde6d-662c-4c21-b82b-bd15902618b6-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:59.556897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.556724    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e4bde6d-662c-4c21-b82b-bd15902618b6-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:45:59.966938 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.966903    2574 generic.go:356] "Generic (PLEG): container finished" podID="1e4bde6d-662c-4c21-b82b-bd15902618b6" containerID="6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c" exitCode=0
Sep 15 11:45:59.967103 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.966951    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"
Sep 15 11:45:59.967103 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.966985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh" event={"ID":"1e4bde6d-662c-4c21-b82b-bd15902618b6","Type":"ContainerDied","Data":"6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c"}
Sep 15 11:45:59.967103 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.967027    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh" event={"ID":"1e4bde6d-662c-4c21-b82b-bd15902618b6","Type":"ContainerDied","Data":"6fe1ba119f6ece917eac4783558f4aaf64876c2f31dfebcb23a0a34dafc3c4c3"}
Sep 15 11:45:59.967103 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.967044    2574 scope.go:117] "RemoveContainer" containerID="6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c"
Sep 15 11:45:59.976006 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.975990    2574 scope.go:117] "RemoveContainer" containerID="6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c"
Sep 15 11:45:59.976246 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:45:59.976224    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c\": container with ID starting with 6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c not found: ID does not exist" containerID="6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c"
Sep 15 11:45:59.976312 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.976258    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c"} err="failed to get container status \"6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c\": rpc error: code = NotFound desc = could not find container \"6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c\": container with ID starting with 6f1981d22e8188b7dbc699d5f3ceb1ed9131b82e65170ebdb38cb3f64f2c001c not found: ID does not exist"
Sep 15 11:45:59.987299 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.987267    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"]
Sep 15 11:45:59.991229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:45:59.991207    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-849a35/noten-model-849a35-kserve-7fc486f495-9qjxh"]
Sep 15 11:46:00.665544 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:00.665493    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4bde6d-662c-4c21-b82b-bd15902618b6" path="/var/lib/kubelet/pods/1e4bde6d-662c-4c21-b82b-bd15902618b6/volumes"
Sep 15 11:46:18.962339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962301    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"]
Sep 15 11:46:18.962838 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962821    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="626d9add-f766-4b85-bea5-8455f4b2882e" containerName="payload-processing"
Sep 15 11:46:18.962838 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962840    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="626d9add-f766-4b85-bea5-8455f4b2882e" containerName="payload-processing"
Sep 15 11:46:18.962914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962854    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58aee0ab-904e-4ff0-8f17-a311d714b22f" containerName="payload-pre-processing"
Sep 15 11:46:18.962914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962860    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="58aee0ab-904e-4ff0-8f17-a311d714b22f" containerName="payload-pre-processing"
Sep 15 11:46:18.962914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962867    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4bde6d-662c-4c21-b82b-bd15902618b6" containerName="main"
Sep 15 11:46:18.962914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962873    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4bde6d-662c-4c21-b82b-bd15902618b6" containerName="main"
Sep 15 11:46:18.963081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962953    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="626d9add-f766-4b85-bea5-8455f4b2882e" containerName="payload-processing"
Sep 15 11:46:18.963081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962961    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="58aee0ab-904e-4ff0-8f17-a311d714b22f" containerName="payload-pre-processing"
Sep 15 11:46:18.963081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.962967    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4bde6d-662c-4c21-b82b-bd15902618b6" containerName="main"
Sep 15 11:46:18.967354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.967338    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:18.969616 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.969597    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-26f1b0-dockercfg-pg5np\""
Sep 15 11:46:18.969890 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.969875    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-26f1b0\""
Sep 15 11:46:18.985775 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.985740    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"]
Sep 15 11:46:18.991545 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.991521    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"]
Sep 15 11:46:18.995110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:18.995089    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.002292 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.002269    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"]
Sep 15 11:46:19.151754 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.151719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ths9\" (UniqueName: \"kubernetes.io/projected/98770b1b-39e6-4466-a789-8f239f84be38-kube-api-access-5ths9\") pod \"payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.151923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.151760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98770b1b-39e6-4466-a789-8f239f84be38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.151923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.151788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a8618-d556-405c-8172-0c718a1d564d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.151923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.151830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmqd9\" (UniqueName: \"kubernetes.io/projected/353a8618-d556-405c-8172-0c718a1d564d-kube-api-access-cmqd9\") pod \"payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.252951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.252869    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ths9\" (UniqueName: \"kubernetes.io/projected/98770b1b-39e6-4466-a789-8f239f84be38-kube-api-access-5ths9\") pod \"payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.252951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.252912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98770b1b-39e6-4466-a789-8f239f84be38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.252951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.252930    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a8618-d556-405c-8172-0c718a1d564d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.252951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.252950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmqd9\" (UniqueName: \"kubernetes.io/projected/353a8618-d556-405c-8172-0c718a1d564d-kube-api-access-cmqd9\") pod \"payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.253657 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.253632    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98770b1b-39e6-4466-a789-8f239f84be38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.253767 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.253676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a8618-d556-405c-8172-0c718a1d564d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.260239 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.260218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmqd9\" (UniqueName: \"kubernetes.io/projected/353a8618-d556-405c-8172-0c718a1d564d-kube-api-access-cmqd9\") pod \"payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.260333 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.260316    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ths9\" (UniqueName: \"kubernetes.io/projected/98770b1b-39e6-4466-a789-8f239f84be38-kube-api-access-5ths9\") pod \"payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.279319 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.279300    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:19.308106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.308077    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:19.552117 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.552082    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"]
Sep 15 11:46:19.553596 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:46:19.553568    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98770b1b_39e6_4466_a789_8f239f84be38.slice/crio-93ca83e49d2352fa447c66046da641aa56253aa6b26d63c7db8244aa2ee0c9fa WatchSource:0}: Error finding container 93ca83e49d2352fa447c66046da641aa56253aa6b26d63c7db8244aa2ee0c9fa: Status 404 returned error can't find the container with id 93ca83e49d2352fa447c66046da641aa56253aa6b26d63c7db8244aa2ee0c9fa
Sep 15 11:46:19.589475 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:19.589449    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"]
Sep 15 11:46:19.590384 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:46:19.590356    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353a8618_d556_405c_8172_0c718a1d564d.slice/crio-42a169410864c738d28726a11d532534e4606a0a4a2a314a67ca4b99bf017015 WatchSource:0}: Error finding container 42a169410864c738d28726a11d532534e4606a0a4a2a314a67ca4b99bf017015: Status 404 returned error can't find the container with id 42a169410864c738d28726a11d532534e4606a0a4a2a314a67ca4b99bf017015
Sep 15 11:46:20.082548 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.082485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw" event={"ID":"353a8618-d556-405c-8172-0c718a1d564d","Type":"ContainerStarted","Data":"37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b"}
Sep 15 11:46:20.083013 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.082569    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw" event={"ID":"353a8618-d556-405c-8172-0c718a1d564d","Type":"ContainerStarted","Data":"42a169410864c738d28726a11d532534e4606a0a4a2a314a67ca4b99bf017015"}
Sep 15 11:46:20.083013 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.082603    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:20.084260 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.084231    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n" event={"ID":"98770b1b-39e6-4466-a789-8f239f84be38","Type":"ContainerStarted","Data":"cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534"}
Sep 15 11:46:20.084387 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.084284    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n" event={"ID":"98770b1b-39e6-4466-a789-8f239f84be38","Type":"ContainerStarted","Data":"93ca83e49d2352fa447c66046da641aa56253aa6b26d63c7db8244aa2ee0c9fa"}
Sep 15 11:46:20.084387 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.084313    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:20.097989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.097949    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw" podStartSLOduration=2.09793371 podStartE2EDuration="2.09793371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:20.097787225 +0000 UTC m=+1273.875463486" watchObservedRunningTime="2026-09-15 11:46:20.09793371 +0000 UTC m=+1273.875609967"
Sep 15 11:46:20.114751 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:20.114696    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n" podStartSLOduration=2.114680506 podStartE2EDuration="2.114680506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:20.112991471 +0000 UTC m=+1273.890667731" watchObservedRunningTime="2026-09-15 11:46:20.114680506 +0000 UTC m=+1273.892356784"
Sep 15 11:46:23.755407 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.755369    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"]
Sep 15 11:46:23.759186 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.759165    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.762240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.762213    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-944c275f\"/\"default-dockercfg-n5f5f\""
Sep 15 11:46:23.762365 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.762217    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-944c275f\"/\"e2e-lifecycle-model-944c275f-kserve-self-signed-certs\""
Sep 15 11:46:23.762365 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.762217    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-944c275f\"/\"openshift-service-ca.crt\""
Sep 15 11:46:23.762365 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.762290    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-944c275f\"/\"kube-root-ca.crt\""
Sep 15 11:46:23.767442 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.767419    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"]
Sep 15 11:46:23.799749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.799722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da7c1920-e262-4033-80da-ca1e2446073d-tls-certs\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.799902 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.799820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-dshm\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.799902 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.799867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7mpg\" (UniqueName: \"kubernetes.io/projected/da7c1920-e262-4033-80da-ca1e2446073d-kube-api-access-t7mpg\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.799980 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.799944    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-home\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.800018 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.800001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-tmp-dir\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.800097 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.800081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-model-cache\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.900646 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.900608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da7c1920-e262-4033-80da-ca1e2446073d-tls-certs\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.900813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.900702    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-dshm\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.900813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.900749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7mpg\" (UniqueName: \"kubernetes.io/projected/da7c1920-e262-4033-80da-ca1e2446073d-kube-api-access-t7mpg\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.900813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.900787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-home\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.900953 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.900820    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-tmp-dir\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.900953 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.900895    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-model-cache\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.901212 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.901181    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-home\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.901212 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.901210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-tmp-dir\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.901383 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.901241    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-model-cache\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.903111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.903086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da7c1920-e262-4033-80da-ca1e2446073d-tls-certs\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.903192 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.903110    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-dshm\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:23.911435 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:23.911417    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7mpg\" (UniqueName: \"kubernetes.io/projected/da7c1920-e262-4033-80da-ca1e2446073d-kube-api-access-t7mpg\") pod \"e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") " pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:24.070441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:24.070358    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:24.344954 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:24.344925    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"]
Sep 15 11:46:24.346480 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:46:24.346451    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda7c1920_e262_4033_80da_ca1e2446073d.slice/crio-951fbee26be76b40616f15cb26df1e419f934fa6e101b6f05913d9361313b0e7 WatchSource:0}: Error finding container 951fbee26be76b40616f15cb26df1e419f934fa6e101b6f05913d9361313b0e7: Status 404 returned error can't find the container with id 951fbee26be76b40616f15cb26df1e419f934fa6e101b6f05913d9361313b0e7
Sep 15 11:46:25.113213 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:25.113173    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6" event={"ID":"da7c1920-e262-4033-80da-ca1e2446073d","Type":"ContainerStarted","Data":"7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1"}
Sep 15 11:46:25.113618 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:25.113225    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6" event={"ID":"da7c1920-e262-4033-80da-ca1e2446073d","Type":"ContainerStarted","Data":"951fbee26be76b40616f15cb26df1e419f934fa6e101b6f05913d9361313b0e7"}
Sep 15 11:46:25.132126 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:25.132077    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6" podStartSLOduration=2.132048657 podStartE2EDuration="2.132048657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:25.130440505 +0000 UTC m=+1278.908116762" watchObservedRunningTime="2026-09-15 11:46:25.132048657 +0000 UTC m=+1278.909724913"
Sep 15 11:46:26.650529 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:26.650478    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"]
Sep 15 11:46:31.091867 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:31.091839    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 11:46:31.092327 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:31.091938    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 11:46:34.070843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:34.070809    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:34.071274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:34.071075    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:34.083725 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:34.083695    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:34.174280 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:34.174252    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:46:41.673740 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:41.673699    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx" podUID="4977d65d-a090-4325-9ac1-4c02ae939e47" containerName="main" containerID="cri-o://6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c" gracePeriod=45
Sep 15 11:46:41.926053 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:41.925995    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:46:42.087591 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.087561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlvw8\" (UniqueName: \"kubernetes.io/projected/4977d65d-a090-4325-9ac1-4c02ae939e47-kube-api-access-wlvw8\") pod \"4977d65d-a090-4325-9ac1-4c02ae939e47\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") "
Sep 15 11:46:42.087765 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.087663    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977d65d-a090-4325-9ac1-4c02ae939e47-tls-certs\") pod \"4977d65d-a090-4325-9ac1-4c02ae939e47\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") "
Sep 15 11:46:42.087765 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.087735    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-tmp-dir\") pod \"4977d65d-a090-4325-9ac1-4c02ae939e47\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") "
Sep 15 11:46:42.087765 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.087751    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-home\") pod \"4977d65d-a090-4325-9ac1-4c02ae939e47\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") "
Sep 15 11:46:42.087910 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.087781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-model-cache\") pod \"4977d65d-a090-4325-9ac1-4c02ae939e47\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") "
Sep 15 11:46:42.087910 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.087817    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-dshm\") pod \"4977d65d-a090-4325-9ac1-4c02ae939e47\" (UID: \"4977d65d-a090-4325-9ac1-4c02ae939e47\") "
Sep 15 11:46:42.088098 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.088072    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-home" (OuterVolumeSpecName: "home") pod "4977d65d-a090-4325-9ac1-4c02ae939e47" (UID: "4977d65d-a090-4325-9ac1-4c02ae939e47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:42.088167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.088085    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4977d65d-a090-4325-9ac1-4c02ae939e47" (UID: "4977d65d-a090-4325-9ac1-4c02ae939e47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:42.088167 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.088098    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-model-cache" (OuterVolumeSpecName: "model-cache") pod "4977d65d-a090-4325-9ac1-4c02ae939e47" (UID: "4977d65d-a090-4325-9ac1-4c02ae939e47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:42.090049 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.090026    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-dshm" (OuterVolumeSpecName: "dshm") pod "4977d65d-a090-4325-9ac1-4c02ae939e47" (UID: "4977d65d-a090-4325-9ac1-4c02ae939e47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:46:42.090219 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.090202    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4977d65d-a090-4325-9ac1-4c02ae939e47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4977d65d-a090-4325-9ac1-4c02ae939e47" (UID: "4977d65d-a090-4325-9ac1-4c02ae939e47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:46:42.090288 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.090221    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4977d65d-a090-4325-9ac1-4c02ae939e47-kube-api-access-wlvw8" (OuterVolumeSpecName: "kube-api-access-wlvw8") pod "4977d65d-a090-4325-9ac1-4c02ae939e47" (UID: "4977d65d-a090-4325-9ac1-4c02ae939e47"). InnerVolumeSpecName "kube-api-access-wlvw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:46:42.189315 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.189226    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4977d65d-a090-4325-9ac1-4c02ae939e47-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.189315 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.189288    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.189315 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.189302    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.189315 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.189315    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.189606 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.189327    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4977d65d-a090-4325-9ac1-4c02ae939e47-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.189606 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.189337    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlvw8\" (UniqueName: \"kubernetes.io/projected/4977d65d-a090-4325-9ac1-4c02ae939e47-kube-api-access-wlvw8\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:46:42.207660 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.207632    2574 generic.go:356] "Generic (PLEG): container finished" podID="4977d65d-a090-4325-9ac1-4c02ae939e47" containerID="6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c" exitCode=0
Sep 15 11:46:42.207844 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.207688    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"
Sep 15 11:46:42.207844 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.207719    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx" event={"ID":"4977d65d-a090-4325-9ac1-4c02ae939e47","Type":"ContainerDied","Data":"6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c"}
Sep 15 11:46:42.207844 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.207777    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx" event={"ID":"4977d65d-a090-4325-9ac1-4c02ae939e47","Type":"ContainerDied","Data":"56979b2f17980c58f1c3ca83c3ae05a90ce143bc2c4b63344d1ff23f8f6c8e6f"}
Sep 15 11:46:42.207844 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.207801    2574 scope.go:117] "RemoveContainer" containerID="6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c"
Sep 15 11:46:42.217945 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.217929    2574 scope.go:117] "RemoveContainer" containerID="6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c"
Sep 15 11:46:42.218233 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:46:42.218212    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c\": container with ID starting with 6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c not found: ID does not exist" containerID="6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c"
Sep 15 11:46:42.218280 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.218241    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c"} err="failed to get container status \"6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c\": rpc error: code = NotFound desc = could not find container \"6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c\": container with ID starting with 6d85af878ccc5a893427d38cb881a1b10028a7c3d413792b03f34ff01282b66c not found: ID does not exist"
Sep 15 11:46:42.228632 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.228613    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"]
Sep 15 11:46:42.232415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.232393    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/e2e-conflict-b-f81d2f5d-kserve-796c7f799c-m22tx"]
Sep 15 11:46:42.665486 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:42.665455    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4977d65d-a090-4325-9ac1-4c02ae939e47" path="/var/lib/kubelet/pods/4977d65d-a090-4325-9ac1-4c02ae939e47/volumes"
Sep 15 11:46:53.656151 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.656117    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"]
Sep 15 11:46:53.656602 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.656589    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4977d65d-a090-4325-9ac1-4c02ae939e47" containerName="main"
Sep 15 11:46:53.656602 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.656604    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4977d65d-a090-4325-9ac1-4c02ae939e47" containerName="main"
Sep 15 11:46:53.656698 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.656690    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4977d65d-a090-4325-9ac1-4c02ae939e47" containerName="main"
Sep 15 11:46:53.661622 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.661602    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.664862 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.664848    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-dd6959-dockercfg-4hdrh\""
Sep 15 11:46:53.664933 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.664852    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-dd6959\""
Sep 15 11:46:53.707380 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.707335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.707588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.707400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmcsr\" (UniqueName: \"kubernetes.io/projected/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-kube-api-access-dmcsr\") pod \"payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.741446 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.741411    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"]
Sep 15 11:46:53.808961 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.808925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmcsr\" (UniqueName: \"kubernetes.io/projected/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-kube-api-access-dmcsr\") pod \"payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.809136 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.809118    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.810024 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.809997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.822149 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.822126    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmcsr\" (UniqueName: \"kubernetes.io/projected/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-kube-api-access-dmcsr\") pod \"payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:53.971818 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:53.971731    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:54.242334 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:54.242308    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"]
Sep 15 11:46:54.243757 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:46:54.243730    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacbe108_cc0d_4108_ad49_cbcf39bfc9cd.slice/crio-a1143c329d8aaf5c6a9b33a8ce82a8e9a8bb72c33e030a900d54176c84d444eb WatchSource:0}: Error finding container a1143c329d8aaf5c6a9b33a8ce82a8e9a8bb72c33e030a900d54176c84d444eb: Status 404 returned error can't find the container with id a1143c329d8aaf5c6a9b33a8ce82a8e9a8bb72c33e030a900d54176c84d444eb
Sep 15 11:46:54.280855 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:54.280806    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2" event={"ID":"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd","Type":"ContainerStarted","Data":"a1143c329d8aaf5c6a9b33a8ce82a8e9a8bb72c33e030a900d54176c84d444eb"}
Sep 15 11:46:55.288855 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:55.288819    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2" event={"ID":"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd","Type":"ContainerStarted","Data":"487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050"}
Sep 15 11:46:55.289256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:55.288884    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:46:55.303554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:46:55.303474    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2" podStartSLOduration=2.303455869 podStartE2EDuration="2.303455869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:46:55.303056764 +0000 UTC m=+1309.080733020" watchObservedRunningTime="2026-09-15 11:46:55.303455869 +0000 UTC m=+1309.081132127"
Sep 15 11:47:06.296720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:06.296681    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 11:47:20.853134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:20.853087    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"]
Sep 15 11:47:20.858383 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:20.858358    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:20.860758 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:20.860729    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:47:20.861050 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:20.861029    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:47:20.863399 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:20.863377    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"]
Sep 15 11:47:20.988860 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:20.988822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0790e845-a25f-4a65-86d5-11f4dfff8290-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-944c275f-8zpdz\" (UID: \"0790e845-a25f-4a65-86d5-11f4dfff8290\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:21.090007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:21.089972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0790e845-a25f-4a65-86d5-11f4dfff8290-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-944c275f-8zpdz\" (UID: \"0790e845-a25f-4a65-86d5-11f4dfff8290\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:21.090594 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:21.090574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0790e845-a25f-4a65-86d5-11f4dfff8290-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-944c275f-8zpdz\" (UID: \"0790e845-a25f-4a65-86d5-11f4dfff8290\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:21.171352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:21.171304    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:21.441051 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:21.441022    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"]
Sep 15 11:47:21.442915 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:47:21.442890    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0790e845_a25f_4a65_86d5_11f4dfff8290.slice/crio-c7fc2a8ce1b2799be6c3c620e6326f693b0ea0612c7b094073abda7241bca5cb WatchSource:0}: Error finding container c7fc2a8ce1b2799be6c3c620e6326f693b0ea0612c7b094073abda7241bca5cb: Status 404 returned error can't find the container with id c7fc2a8ce1b2799be6c3c620e6326f693b0ea0612c7b094073abda7241bca5cb
Sep 15 11:47:22.448637 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:22.448603    2574 generic.go:356] "Generic (PLEG): container finished" podID="0790e845-a25f-4a65-86d5-11f4dfff8290" containerID="37f57e885715691f773e2e5450d7eaf862ef4ba289f793718a1701ea7df67c5d" exitCode=0
Sep 15 11:47:22.449077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:22.448690    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz" event={"ID":"0790e845-a25f-4a65-86d5-11f4dfff8290","Type":"ContainerDied","Data":"37f57e885715691f773e2e5450d7eaf862ef4ba289f793718a1701ea7df67c5d"}
Sep 15 11:47:22.449077 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:22.448743    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz" event={"ID":"0790e845-a25f-4a65-86d5-11f4dfff8290","Type":"ContainerStarted","Data":"c7fc2a8ce1b2799be6c3c620e6326f693b0ea0612c7b094073abda7241bca5cb"}
Sep 15 11:47:23.585898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:23.585872    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:23.718188 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:23.718108    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0790e845-a25f-4a65-86d5-11f4dfff8290-maas-api-service-ca\") pod \"0790e845-a25f-4a65-86d5-11f4dfff8290\" (UID: \"0790e845-a25f-4a65-86d5-11f4dfff8290\") "
Sep 15 11:47:23.718458 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:23.718435    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0790e845-a25f-4a65-86d5-11f4dfff8290-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0790e845-a25f-4a65-86d5-11f4dfff8290" (UID: "0790e845-a25f-4a65-86d5-11f4dfff8290"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:47:23.819009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:23.818975    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0790e845-a25f-4a65-86d5-11f4dfff8290-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:47:24.461009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:24.460970    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz" event={"ID":"0790e845-a25f-4a65-86d5-11f4dfff8290","Type":"ContainerDied","Data":"c7fc2a8ce1b2799be6c3c620e6326f693b0ea0612c7b094073abda7241bca5cb"}
Sep 15 11:47:24.461009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:24.461009    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"
Sep 15 11:47:24.461224 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:24.461032    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7fc2a8ce1b2799be6c3c620e6326f693b0ea0612c7b094073abda7241bca5cb"
Sep 15 11:47:36.625607 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.625521    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"]
Sep 15 11:47:36.626091 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.625793    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7" podUID="3a87897e-09e3-4969-bfd1-0d2f52080641" containerName="payload-processing" containerID="cri-o://bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f" gracePeriod=30
Sep 15 11:47:36.634869 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.633058    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"]
Sep 15 11:47:36.634869 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.633309    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k" podUID="7cf34740-a9cf-4d58-b8c6-bae94076ba66" containerName="payload-pre-processing" containerID="cri-o://f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1" gracePeriod=30
Sep 15 11:47:36.793051 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.792999    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"]
Sep 15 11:47:36.798227 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.798183    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-944c275f-8zpdz"]
Sep 15 11:47:36.915629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.915605    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:47:36.918756 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.918731    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:47:36.957461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.957429    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkrx7\" (UniqueName: \"kubernetes.io/projected/7cf34740-a9cf-4d58-b8c6-bae94076ba66-kube-api-access-tkrx7\") pod \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") "
Sep 15 11:47:36.957659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.957475    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf34740-a9cf-4d58-b8c6-bae94076ba66-plugins-config-volume\") pod \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\" (UID: \"7cf34740-a9cf-4d58-b8c6-bae94076ba66\") "
Sep 15 11:47:36.957659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.957560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a87897e-09e3-4969-bfd1-0d2f52080641-plugins-config-volume\") pod \"3a87897e-09e3-4969-bfd1-0d2f52080641\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") "
Sep 15 11:47:36.957659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.957642    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4dkn\" (UniqueName: \"kubernetes.io/projected/3a87897e-09e3-4969-bfd1-0d2f52080641-kube-api-access-s4dkn\") pod \"3a87897e-09e3-4969-bfd1-0d2f52080641\" (UID: \"3a87897e-09e3-4969-bfd1-0d2f52080641\") "
Sep 15 11:47:36.957988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.957955    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a87897e-09e3-4969-bfd1-0d2f52080641-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a87897e-09e3-4969-bfd1-0d2f52080641" (UID: "3a87897e-09e3-4969-bfd1-0d2f52080641"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:47:36.957988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.957958    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf34740-a9cf-4d58-b8c6-bae94076ba66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7cf34740-a9cf-4d58-b8c6-bae94076ba66" (UID: "7cf34740-a9cf-4d58-b8c6-bae94076ba66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:47:36.958163 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.958060    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cf34740-a9cf-4d58-b8c6-bae94076ba66-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:47:36.958163 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.958081    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a87897e-09e3-4969-bfd1-0d2f52080641-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:47:36.959673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.959654    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf34740-a9cf-4d58-b8c6-bae94076ba66-kube-api-access-tkrx7" (OuterVolumeSpecName: "kube-api-access-tkrx7") pod "7cf34740-a9cf-4d58-b8c6-bae94076ba66" (UID: "7cf34740-a9cf-4d58-b8c6-bae94076ba66"). InnerVolumeSpecName "kube-api-access-tkrx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:47:36.960196 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:36.960169    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a87897e-09e3-4969-bfd1-0d2f52080641-kube-api-access-s4dkn" (OuterVolumeSpecName: "kube-api-access-s4dkn") pod "3a87897e-09e3-4969-bfd1-0d2f52080641" (UID: "3a87897e-09e3-4969-bfd1-0d2f52080641"). InnerVolumeSpecName "kube-api-access-s4dkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:47:37.059056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.059019    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4dkn\" (UniqueName: \"kubernetes.io/projected/3a87897e-09e3-4969-bfd1-0d2f52080641-kube-api-access-s4dkn\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:47:37.059056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.059047    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkrx7\" (UniqueName: \"kubernetes.io/projected/7cf34740-a9cf-4d58-b8c6-bae94076ba66-kube-api-access-tkrx7\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:47:37.534242 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.534206    2574 generic.go:356] "Generic (PLEG): container finished" podID="7cf34740-a9cf-4d58-b8c6-bae94076ba66" containerID="f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1" exitCode=0
Sep 15 11:47:37.534424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.534251    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"
Sep 15 11:47:37.534424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.534296    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k" event={"ID":"7cf34740-a9cf-4d58-b8c6-bae94076ba66","Type":"ContainerDied","Data":"f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1"}
Sep 15 11:47:37.534424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.534360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k" event={"ID":"7cf34740-a9cf-4d58-b8c6-bae94076ba66","Type":"ContainerDied","Data":"066964c388c481323f3013b04d05c67c2ca094d7d01c1924707ab40b62f374b2"}
Sep 15 11:47:37.534424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.534390    2574 scope.go:117] "RemoveContainer" containerID="f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1"
Sep 15 11:47:37.535615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.535538    2574 generic.go:356] "Generic (PLEG): container finished" podID="3a87897e-09e3-4969-bfd1-0d2f52080641" containerID="bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f" exitCode=0
Sep 15 11:47:37.535615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.535569    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7" event={"ID":"3a87897e-09e3-4969-bfd1-0d2f52080641","Type":"ContainerDied","Data":"bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f"}
Sep 15 11:47:37.535615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.535579    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"
Sep 15 11:47:37.535615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.535608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7" event={"ID":"3a87897e-09e3-4969-bfd1-0d2f52080641","Type":"ContainerDied","Data":"20486d6fe67f5cf16c5a3c7f9e95c83e0120903a7157114bd5e7b194ee889467"}
Sep 15 11:47:37.544608 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.544588    2574 scope.go:117] "RemoveContainer" containerID="f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1"
Sep 15 11:47:37.544909 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:47:37.544888    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1\": container with ID starting with f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1 not found: ID does not exist" containerID="f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1"
Sep 15 11:47:37.545001 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.544915    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1"} err="failed to get container status \"f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1\": rpc error: code = NotFound desc = could not find container \"f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1\": container with ID starting with f8d1effe7851e0ca8786330957568595f31dd10533782d8673439ecff571f2c1 not found: ID does not exist"
Sep 15 11:47:37.545001 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.544934    2574 scope.go:117] "RemoveContainer" containerID="bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f"
Sep 15 11:47:37.553365 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.553350    2574 scope.go:117] "RemoveContainer" containerID="bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f"
Sep 15 11:47:37.553631 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:47:37.553611    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f\": container with ID starting with bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f not found: ID does not exist" containerID="bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f"
Sep 15 11:47:37.553699 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.553637    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f"} err="failed to get container status \"bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f\": rpc error: code = NotFound desc = could not find container \"bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f\": container with ID starting with bdff3ede53a46b241df07495d3f5885fb003ef426e1f14f61fa2d6fb5b478e6f not found: ID does not exist"
Sep 15 11:47:37.560254 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.560229    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"]
Sep 15 11:47:37.562378 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.562360    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-944c275f-7587564f9c-gz7h7"]
Sep 15 11:47:37.571563 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.571545    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"]
Sep 15 11:47:37.575024 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:37.575006    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-944c275f-dc4cbd886-tnv9k"]
Sep 15 11:47:38.665758 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:38.665725    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0790e845-a25f-4a65-86d5-11f4dfff8290" path="/var/lib/kubelet/pods/0790e845-a25f-4a65-86d5-11f4dfff8290/volumes"
Sep 15 11:47:38.666141 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:38.666126    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a87897e-09e3-4969-bfd1-0d2f52080641" path="/var/lib/kubelet/pods/3a87897e-09e3-4969-bfd1-0d2f52080641/volumes"
Sep 15 11:47:38.666460 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:38.666437    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cf34740-a9cf-4d58-b8c6-bae94076ba66" path="/var/lib/kubelet/pods/7cf34740-a9cf-4d58-b8c6-bae94076ba66/volumes"
Sep 15 11:47:44.888930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:44.888897    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"]
Sep 15 11:47:49.137887 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.137852    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"]
Sep 15 11:47:49.138339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138325    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf34740-a9cf-4d58-b8c6-bae94076ba66" containerName="payload-pre-processing"
Sep 15 11:47:49.138339 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138340    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf34740-a9cf-4d58-b8c6-bae94076ba66" containerName="payload-pre-processing"
Sep 15 11:47:49.138416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138359    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0790e845-a25f-4a65-86d5-11f4dfff8290" containerName="revoke-keys"
Sep 15 11:47:49.138416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138366    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0790e845-a25f-4a65-86d5-11f4dfff8290" containerName="revoke-keys"
Sep 15 11:47:49.138416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138377    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a87897e-09e3-4969-bfd1-0d2f52080641" containerName="payload-processing"
Sep 15 11:47:49.138416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138383    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a87897e-09e3-4969-bfd1-0d2f52080641" containerName="payload-processing"
Sep 15 11:47:49.138580 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138466    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0790e845-a25f-4a65-86d5-11f4dfff8290" containerName="revoke-keys"
Sep 15 11:47:49.138580 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138476    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a87897e-09e3-4969-bfd1-0d2f52080641" containerName="payload-processing"
Sep 15 11:47:49.138580 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.138485    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cf34740-a9cf-4d58-b8c6-bae94076ba66" containerName="payload-pre-processing"
Sep 15 11:47:49.143138 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.143119    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.147427 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.147405    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-26f1b0\"/\"kube-root-ca.crt\""
Sep 15 11:47:49.148022 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.148003    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-26f1b0\"/\"default-dockercfg-wm2v2\""
Sep 15 11:47:49.149274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.149255    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-26f1b0\"/\"auth-test-model-26f1b0-kserve-self-signed-certs\""
Sep 15 11:47:49.150464 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.150446    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-26f1b0\"/\"openshift-service-ca.crt\""
Sep 15 11:47:49.159880 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.159860    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"]
Sep 15 11:47:49.286816 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.286778    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-home\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.286816 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.286817    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsq6\" (UniqueName: \"kubernetes.io/projected/55cc7119-2a52-4e2a-beed-174880730b72-kube-api-access-6wsq6\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.287059 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.286849    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc7119-2a52-4e2a-beed-174880730b72-tls-certs\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.287059 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.286941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-tmp-dir\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.287059 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.287006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-dshm\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.287211 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.287077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-model-cache\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.387865 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.387827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-dshm\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.387874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-model-cache\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.387919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-home\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.387935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsq6\" (UniqueName: \"kubernetes.io/projected/55cc7119-2a52-4e2a-beed-174880730b72-kube-api-access-6wsq6\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.387962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc7119-2a52-4e2a-beed-174880730b72-tls-certs\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.388025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-tmp-dir\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388367 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.388341    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-model-cache\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388439 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.388393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-home\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.388491 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.388437    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-tmp-dir\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.390157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.390132    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-dshm\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.390420 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.390394    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc7119-2a52-4e2a-beed-174880730b72-tls-certs\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.396086 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.396069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsq6\" (UniqueName: \"kubernetes.io/projected/55cc7119-2a52-4e2a-beed-174880730b72-kube-api-access-6wsq6\") pod \"auth-test-model-26f1b0-kserve-548dd7687-g4n84\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") " pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.452982 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.452946    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:49.723486 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:49.723461    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"]
Sep 15 11:47:49.725248 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:47:49.725226    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cc7119_2a52_4e2a_beed_174880730b72.slice/crio-99d03ac659410ba0cfd0d51ad35f4a7b1b7057a334bbf32c79a9c1c787b10d50 WatchSource:0}: Error finding container 99d03ac659410ba0cfd0d51ad35f4a7b1b7057a334bbf32c79a9c1c787b10d50: Status 404 returned error can't find the container with id 99d03ac659410ba0cfd0d51ad35f4a7b1b7057a334bbf32c79a9c1c787b10d50
Sep 15 11:47:50.622054 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:50.622017    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84" event={"ID":"55cc7119-2a52-4e2a-beed-174880730b72","Type":"ContainerStarted","Data":"54ced14f2b02b146330a0e3a5587271482bcabeb97382648ba22be6972f5734f"}
Sep 15 11:47:50.622054 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:50.622065    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84" event={"ID":"55cc7119-2a52-4e2a-beed-174880730b72","Type":"ContainerStarted","Data":"99d03ac659410ba0cfd0d51ad35f4a7b1b7057a334bbf32c79a9c1c787b10d50"}
Sep 15 11:47:50.639927 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:50.639881    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84" podStartSLOduration=1.639865124 podStartE2EDuration="1.639865124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:47:50.638100528 +0000 UTC m=+1364.415776788" watchObservedRunningTime="2026-09-15 11:47:50.639865124 +0000 UTC m=+1364.417541381"
Sep 15 11:47:59.453448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:59.453405    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:59.454027 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:59.453466    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:59.466300 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:59.466275    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:59.683746 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:59.683719    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:47:59.909492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:47:59.909334    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6" podUID="da7c1920-e262-4033-80da-ca1e2446073d" containerName="main" containerID="cri-o://7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1" gracePeriod=45
Sep 15 11:48:00.161914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.161848    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:48:00.192852 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.192819    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-model-cache\") pod \"da7c1920-e262-4033-80da-ca1e2446073d\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") "
Sep 15 11:48:00.193018 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.192873    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da7c1920-e262-4033-80da-ca1e2446073d-tls-certs\") pod \"da7c1920-e262-4033-80da-ca1e2446073d\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") "
Sep 15 11:48:00.193018 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.192958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-tmp-dir\") pod \"da7c1920-e262-4033-80da-ca1e2446073d\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") "
Sep 15 11:48:00.193018 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193012    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7mpg\" (UniqueName: \"kubernetes.io/projected/da7c1920-e262-4033-80da-ca1e2446073d-kube-api-access-t7mpg\") pod \"da7c1920-e262-4033-80da-ca1e2446073d\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") "
Sep 15 11:48:00.193192 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193048    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-dshm\") pod \"da7c1920-e262-4033-80da-ca1e2446073d\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") "
Sep 15 11:48:00.193192 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193090    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-home\") pod \"da7c1920-e262-4033-80da-ca1e2446073d\" (UID: \"da7c1920-e262-4033-80da-ca1e2446073d\") "
Sep 15 11:48:00.193192 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193088    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-model-cache" (OuterVolumeSpecName: "model-cache") pod "da7c1920-e262-4033-80da-ca1e2446073d" (UID: "da7c1920-e262-4033-80da-ca1e2446073d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:48:00.193355 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193281    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "da7c1920-e262-4033-80da-ca1e2446073d" (UID: "da7c1920-e262-4033-80da-ca1e2446073d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:48:00.193492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193472    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:48:00.193492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193483    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-home" (OuterVolumeSpecName: "home") pod "da7c1920-e262-4033-80da-ca1e2446073d" (UID: "da7c1920-e262-4033-80da-ca1e2446073d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:48:00.193630 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.193520    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:48:00.195700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.195675    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-dshm" (OuterVolumeSpecName: "dshm") pod "da7c1920-e262-4033-80da-ca1e2446073d" (UID: "da7c1920-e262-4033-80da-ca1e2446073d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:48:00.195803 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.195707    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7c1920-e262-4033-80da-ca1e2446073d-kube-api-access-t7mpg" (OuterVolumeSpecName: "kube-api-access-t7mpg") pod "da7c1920-e262-4033-80da-ca1e2446073d" (UID: "da7c1920-e262-4033-80da-ca1e2446073d"). InnerVolumeSpecName "kube-api-access-t7mpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:48:00.195803 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.195754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7c1920-e262-4033-80da-ca1e2446073d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "da7c1920-e262-4033-80da-ca1e2446073d" (UID: "da7c1920-e262-4033-80da-ca1e2446073d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:48:00.294058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.294005    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da7c1920-e262-4033-80da-ca1e2446073d-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:48:00.294058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.294050    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7mpg\" (UniqueName: \"kubernetes.io/projected/da7c1920-e262-4033-80da-ca1e2446073d-kube-api-access-t7mpg\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:48:00.294058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.294060    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:48:00.294058 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.294068    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da7c1920-e262-4033-80da-ca1e2446073d-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:48:00.677851 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.677821    2574 generic.go:356] "Generic (PLEG): container finished" podID="da7c1920-e262-4033-80da-ca1e2446073d" containerID="7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1" exitCode=0
Sep 15 11:48:00.678231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.677860    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"
Sep 15 11:48:00.678231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.677895    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6" event={"ID":"da7c1920-e262-4033-80da-ca1e2446073d","Type":"ContainerDied","Data":"7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1"}
Sep 15 11:48:00.678231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.677936    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6" event={"ID":"da7c1920-e262-4033-80da-ca1e2446073d","Type":"ContainerDied","Data":"951fbee26be76b40616f15cb26df1e419f934fa6e101b6f05913d9361313b0e7"}
Sep 15 11:48:00.678231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.677953    2574 scope.go:117] "RemoveContainer" containerID="7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1"
Sep 15 11:48:00.689756 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.689737    2574 scope.go:117] "RemoveContainer" containerID="7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1"
Sep 15 11:48:00.690028 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:48:00.690006    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1\": container with ID starting with 7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1 not found: ID does not exist" containerID="7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1"
Sep 15 11:48:00.690102 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.690033    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1"} err="failed to get container status \"7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1\": rpc error: code = NotFound desc = could not find container \"7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1\": container with ID starting with 7f38b5b7dfb82ed156312051e626af09f3c0f684cef63724fb181af68ed57be1 not found: ID does not exist"
Sep 15 11:48:00.694432 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.694408    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"]
Sep 15 11:48:00.698352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:00.698332    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-944c275f/e2e-lifecycle-model-944c275f-kserve-5c57698b99-pqnw6"]
Sep 15 11:48:02.666517 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:02.666456    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da7c1920-e262-4033-80da-ca1e2446073d" path="/var/lib/kubelet/pods/da7c1920-e262-4033-80da-ca1e2446073d/volumes"
Sep 15 11:48:25.663815 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.663783    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"]
Sep 15 11:48:25.664260 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.664246    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da7c1920-e262-4033-80da-ca1e2446073d" containerName="main"
Sep 15 11:48:25.664260 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.664261    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="da7c1920-e262-4033-80da-ca1e2446073d" containerName="main"
Sep 15 11:48:25.664369 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.664360    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="da7c1920-e262-4033-80da-ca1e2446073d" containerName="main"
Sep 15 11:48:25.669139 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.669119    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.671883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.671855    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2fab1f\"/\"distinct-2-w4-2fab1f-kserve-self-signed-certs\""
Sep 15 11:48:25.677883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.677846    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"]
Sep 15 11:48:25.756297 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.756258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-tmp-dir\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.756448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.756319    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgzxq\" (UniqueName: \"kubernetes.io/projected/666e4036-8df4-40b8-8937-506770e4af1f-kube-api-access-qgzxq\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.756448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.756342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666e4036-8df4-40b8-8937-506770e4af1f-tls-certs\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.756448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.756363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-model-cache\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.756448 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.756432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-dshm\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.756646 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.756487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-home\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.857860 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.857821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-dshm\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.857891    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-home\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.857980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-tmp-dir\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.858042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgzxq\" (UniqueName: \"kubernetes.io/projected/666e4036-8df4-40b8-8937-506770e4af1f-kube-api-access-qgzxq\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.858074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666e4036-8df4-40b8-8937-506770e4af1f-tls-certs\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.858107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-model-cache\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.858344    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-tmp-dir\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.858361    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-home\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.858600 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.858457    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-model-cache\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.860241 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.860218    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-dshm\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.860657 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.860638    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666e4036-8df4-40b8-8937-506770e4af1f-tls-certs\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.865996 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.865972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgzxq\" (UniqueName: \"kubernetes.io/projected/666e4036-8df4-40b8-8937-506770e4af1f-kube-api-access-qgzxq\") pod \"distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") " pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:25.980488 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:25.980400    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:26.255038 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:26.255007    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"]
Sep 15 11:48:26.256971 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:48:26.256936    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod666e4036_8df4_40b8_8937_506770e4af1f.slice/crio-11d75527e1f4fe397957c6dec137835aac0348c47d112dacf47e467d1540392a WatchSource:0}: Error finding container 11d75527e1f4fe397957c6dec137835aac0348c47d112dacf47e467d1540392a: Status 404 returned error can't find the container with id 11d75527e1f4fe397957c6dec137835aac0348c47d112dacf47e467d1540392a
Sep 15 11:48:26.818573 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:26.818537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx" event={"ID":"666e4036-8df4-40b8-8937-506770e4af1f","Type":"ContainerStarted","Data":"8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465"}
Sep 15 11:48:26.818955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:26.818583    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx" event={"ID":"666e4036-8df4-40b8-8937-506770e4af1f","Type":"ContainerStarted","Data":"11d75527e1f4fe397957c6dec137835aac0348c47d112dacf47e467d1540392a"}
Sep 15 11:48:26.837329 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:26.837275    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx" podStartSLOduration=1.837259126 podStartE2EDuration="1.837259126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:26.835343754 +0000 UTC m=+1400.613020013" watchObservedRunningTime="2026-09-15 11:48:26.837259126 +0000 UTC m=+1400.614935383"
Sep 15 11:48:33.763612 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.763580    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"]
Sep 15 11:48:33.767547 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.767529    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.770798 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.770772    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"auth-test-model-dd6959-kserve-self-signed-certs\""
Sep 15 11:48:33.770798 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.770794    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"openshift-service-ca.crt\""
Sep 15 11:48:33.770975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.770795    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"default-dockercfg-2gsgm\""
Sep 15 11:48:33.770975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.770772    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"kube-root-ca.crt\""
Sep 15 11:48:33.778381 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.778354    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"]
Sep 15 11:48:33.844489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.844460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-model-cache\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.844489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.844508    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7c146-09e5-424f-ad45-f241779f893b-tls-certs\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.844705 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.844579    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-tmp-dir\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.844705 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.844606    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjkr\" (UniqueName: \"kubernetes.io/projected/daa7c146-09e5-424f-ad45-f241779f893b-kube-api-access-tjjkr\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.844705 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.844649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-dshm\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.844705 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.844664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-home\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-model-cache\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7c146-09e5-424f-ad45-f241779f893b-tls-certs\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-tmp-dir\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjkr\" (UniqueName: \"kubernetes.io/projected/daa7c146-09e5-424f-ad45-f241779f893b-kube-api-access-tjjkr\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-dshm\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945729 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-home\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945944 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945878    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-model-cache\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.945944 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945887    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-tmp-dir\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.946026 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.945984    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-home\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.947814 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.947796    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-dshm\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.948123 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.948103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7c146-09e5-424f-ad45-f241779f893b-tls-certs\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:33.959275 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:33.959252    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjkr\" (UniqueName: \"kubernetes.io/projected/daa7c146-09e5-424f-ad45-f241779f893b-kube-api-access-tjjkr\") pod \"auth-test-model-dd6959-kserve-7b78d764f5-hrd5q\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:34.081011 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:34.080922    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:34.353662 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:34.353634    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"]
Sep 15 11:48:34.355162 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:48:34.355136    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaa7c146_09e5_424f_ad45_f241779f893b.slice/crio-198f3f936e45f5c221715598fbeec8657388983cb102215d0fedbdb12e6bf244 WatchSource:0}: Error finding container 198f3f936e45f5c221715598fbeec8657388983cb102215d0fedbdb12e6bf244: Status 404 returned error can't find the container with id 198f3f936e45f5c221715598fbeec8657388983cb102215d0fedbdb12e6bf244
Sep 15 11:48:34.865677 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:34.865640    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q" event={"ID":"daa7c146-09e5-424f-ad45-f241779f893b","Type":"ContainerStarted","Data":"16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4"}
Sep 15 11:48:34.866033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:34.865684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q" event={"ID":"daa7c146-09e5-424f-ad45-f241779f893b","Type":"ContainerStarted","Data":"198f3f936e45f5c221715598fbeec8657388983cb102215d0fedbdb12e6bf244"}
Sep 15 11:48:34.886441 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:34.886388    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q" podStartSLOduration=1.8863711859999999 podStartE2EDuration="1.886371186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:48:34.883375774 +0000 UTC m=+1408.661052031" watchObservedRunningTime="2026-09-15 11:48:34.886371186 +0000 UTC m=+1408.664047444"
Sep 15 11:48:35.981349 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:35.981313    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:35.981756 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:35.981362    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:35.993668 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:35.993640    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:36.888512 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:36.888470    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:48:44.081910 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:44.081871    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:44.082292 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:44.081957    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:44.095260 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:44.095230    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:48:44.939170 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:48:44.939146    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:49:17.565685 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.565601    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"]
Sep 15 11:49:17.569392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.569371    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.571591 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.571565    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9bd868\"/\"embedding-w0-9bd868-kserve-self-signed-certs\""
Sep 15 11:49:17.576924 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.576902    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"]
Sep 15 11:49:17.597429 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.597387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-model-cache\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.597597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.597462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tmp-dir\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.597597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.597520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tls-certs\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.597597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.597562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-dshm\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.597597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.597587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-home\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.597735 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.597670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pth9m\" (UniqueName: \"kubernetes.io/projected/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-kube-api-access-pth9m\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.699104 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.698337    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pth9m\" (UniqueName: \"kubernetes.io/projected/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-kube-api-access-pth9m\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.699576 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.699536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-model-cache\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.699805 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.699790    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tmp-dir\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.699969 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.699948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tls-certs\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.700133 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.700111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-dshm\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.700257 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.700244    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-home\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.700943 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.700923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-home\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.701394 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.701364    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-model-cache\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.701854 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.701837    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tmp-dir\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.705651 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.705629    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tls-certs\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.708215 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.708192    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pth9m\" (UniqueName: \"kubernetes.io/projected/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-kube-api-access-pth9m\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.708563 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.708542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-dshm\") pod \"embedding-w0-9bd868-kserve-76f799d4f9-xg2qn\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") " pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:17.882857 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:17.882755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:18.156864 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:18.156827    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"]
Sep 15 11:49:18.158309 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:49:18.158281    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab82c0c1_eb94_44e9_967f_e1e0ab56f22f.slice/crio-8507711f3e741a8ad26b020da80103113ba5134bf82a354df6a98a49da6c5fe1 WatchSource:0}: Error finding container 8507711f3e741a8ad26b020da80103113ba5134bf82a354df6a98a49da6c5fe1: Status 404 returned error can't find the container with id 8507711f3e741a8ad26b020da80103113ba5134bf82a354df6a98a49da6c5fe1
Sep 15 11:49:19.133521 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:19.133457    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn" event={"ID":"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f","Type":"ContainerStarted","Data":"660619d1e5532e6407bf3ebaf9de94977d74b81c05926eb36e7f85afed6e2497"}
Sep 15 11:49:19.133893 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:19.133536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn" event={"ID":"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f","Type":"ContainerStarted","Data":"8507711f3e741a8ad26b020da80103113ba5134bf82a354df6a98a49da6c5fe1"}
Sep 15 11:49:19.152390 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:19.152341    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn" podStartSLOduration=2.152326764 podStartE2EDuration="2.152326764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:19.150683243 +0000 UTC m=+1452.928359503" watchObservedRunningTime="2026-09-15 11:49:19.152326764 +0000 UTC m=+1452.930003021"
Sep 15 11:49:27.883674 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:27.883634    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:27.884110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:27.883711    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:27.896550 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:27.896530    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:28.193701 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:28.193668    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:49:45.852551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.852489    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"]
Sep 15 11:49:45.856281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.856261    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:45.858951 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.858923    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-01df46\""
Sep 15 11:49:45.859101 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.859079    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-01df46-dockercfg-zfr5v\""
Sep 15 11:49:45.868819 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.868793    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"]
Sep 15 11:49:45.888615 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.888578    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjmp\" (UniqueName: \"kubernetes.io/projected/488f3620-cb7b-4a01-9520-4f8ec4e4a605-kube-api-access-6xjmp\") pod \"payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") " pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:45.888780 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.888694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488f3620-cb7b-4a01-9520-4f8ec4e4a605-plugins-config-volume\") pod \"payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") " pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:45.990053 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.990016    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjmp\" (UniqueName: \"kubernetes.io/projected/488f3620-cb7b-4a01-9520-4f8ec4e4a605-kube-api-access-6xjmp\") pod \"payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") " pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:45.990243 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.990099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488f3620-cb7b-4a01-9520-4f8ec4e4a605-plugins-config-volume\") pod \"payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") " pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:45.990810 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.990789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488f3620-cb7b-4a01-9520-4f8ec4e4a605-plugins-config-volume\") pod \"payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") " pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:45.997965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:45.997937    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjmp\" (UniqueName: \"kubernetes.io/projected/488f3620-cb7b-4a01-9520-4f8ec4e4a605-kube-api-access-6xjmp\") pod \"payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") " pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:46.168947 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:46.168905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:46.443738 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:46.443707    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"]
Sep 15 11:49:46.445084 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:49:46.445057    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488f3620_cb7b_4a01_9520_4f8ec4e4a605.slice/crio-cc84302071049067c6ccb7f2c26d70e6f1bb2f9e7c84db44bae2fa17c455646b WatchSource:0}: Error finding container cc84302071049067c6ccb7f2c26d70e6f1bb2f9e7c84db44bae2fa17c455646b: Status 404 returned error can't find the container with id cc84302071049067c6ccb7f2c26d70e6f1bb2f9e7c84db44bae2fa17c455646b
Sep 15 11:49:47.288965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:47.288928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b" event={"ID":"488f3620-cb7b-4a01-9520-4f8ec4e4a605","Type":"ContainerStarted","Data":"36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d"}
Sep 15 11:49:47.289376 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:47.288980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b" event={"ID":"488f3620-cb7b-4a01-9520-4f8ec4e4a605","Type":"ContainerStarted","Data":"cc84302071049067c6ccb7f2c26d70e6f1bb2f9e7c84db44bae2fa17c455646b"}
Sep 15 11:49:47.289376 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:47.289120    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:49:47.307901 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:47.307845    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b" podStartSLOduration=2.307823416 podStartE2EDuration="2.307823416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:49:47.305793102 +0000 UTC m=+1481.083469395" watchObservedRunningTime="2026-09-15 11:49:47.307823416 +0000 UTC m=+1481.085499677"
Sep 15 11:49:58.297191 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:49:58.297161    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:50:06.728597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:06.728570    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:50:06.732487 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:06.732461    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:50:06.733270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:06.733253    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:50:06.736551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:06.736533    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:50:48.267008 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.266969    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"]
Sep 15 11:50:48.270902 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.270884    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:48.273242 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.273219    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:50:48.273359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.273301    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:50:48.280161 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.280136    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"]
Sep 15 11:50:48.450557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.450492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c7b6a9-76b0-494c-b4db-62d0de2e9325-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01df46-mcj7x\" (UID: \"f9c7b6a9-76b0-494c-b4db-62d0de2e9325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:48.551623 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.551529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c7b6a9-76b0-494c-b4db-62d0de2e9325-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01df46-mcj7x\" (UID: \"f9c7b6a9-76b0-494c-b4db-62d0de2e9325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:48.552147 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.552126    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c7b6a9-76b0-494c-b4db-62d0de2e9325-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-01df46-mcj7x\" (UID: \"f9c7b6a9-76b0-494c-b4db-62d0de2e9325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:48.582579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.582550    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:48.857688 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:48.857662    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"]
Sep 15 11:50:48.859020 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:50:48.858993    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c7b6a9_76b0_494c_b4db_62d0de2e9325.slice/crio-2e922e3a62339bbd6f83cbc66c76bf8187161f3dcb76b8cae4f7cc7de55de844 WatchSource:0}: Error finding container 2e922e3a62339bbd6f83cbc66c76bf8187161f3dcb76b8cae4f7cc7de55de844: Status 404 returned error can't find the container with id 2e922e3a62339bbd6f83cbc66c76bf8187161f3dcb76b8cae4f7cc7de55de844
Sep 15 11:50:49.650512 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:49.650474    2574 generic.go:356] "Generic (PLEG): container finished" podID="f9c7b6a9-76b0-494c-b4db-62d0de2e9325" containerID="7ff78e795b30d9055a6ac2503abebc7f5fe058bf3e77de35376b7e9a2da261a8" exitCode=0
Sep 15 11:50:49.650855 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:49.650563    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x" event={"ID":"f9c7b6a9-76b0-494c-b4db-62d0de2e9325","Type":"ContainerDied","Data":"7ff78e795b30d9055a6ac2503abebc7f5fe058bf3e77de35376b7e9a2da261a8"}
Sep 15 11:50:49.650855 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:49.650617    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x" event={"ID":"f9c7b6a9-76b0-494c-b4db-62d0de2e9325","Type":"ContainerStarted","Data":"2e922e3a62339bbd6f83cbc66c76bf8187161f3dcb76b8cae4f7cc7de55de844"}
Sep 15 11:50:50.807269 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:50.807242    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:50.973826 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:50.973741    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c7b6a9-76b0-494c-b4db-62d0de2e9325-maas-api-service-ca\") pod \"f9c7b6a9-76b0-494c-b4db-62d0de2e9325\" (UID: \"f9c7b6a9-76b0-494c-b4db-62d0de2e9325\") "
Sep 15 11:50:50.974099 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:50.974077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c7b6a9-76b0-494c-b4db-62d0de2e9325-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9c7b6a9-76b0-494c-b4db-62d0de2e9325" (UID: "f9c7b6a9-76b0-494c-b4db-62d0de2e9325"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:50:51.075579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:51.075549    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9c7b6a9-76b0-494c-b4db-62d0de2e9325-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:50:51.662395 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:51.662369    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"
Sep 15 11:50:51.662584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:51.662414    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x" event={"ID":"f9c7b6a9-76b0-494c-b4db-62d0de2e9325","Type":"ContainerDied","Data":"2e922e3a62339bbd6f83cbc66c76bf8187161f3dcb76b8cae4f7cc7de55de844"}
Sep 15 11:50:51.662584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:50:51.662460    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e922e3a62339bbd6f83cbc66c76bf8187161f3dcb76b8cae4f7cc7de55de844"
Sep 15 11:51:04.091830 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.091795    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"]
Sep 15 11:51:04.094458 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.092090    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b" podUID="488f3620-cb7b-4a01-9520-4f8ec4e4a605" containerName="payload-processing" containerID="cri-o://36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d" gracePeriod=30
Sep 15 11:51:04.255588 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.255545    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"]
Sep 15 11:51:04.257645 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.257620    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-01df46-mcj7x"]
Sep 15 11:51:04.361223 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.361195    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:51:04.407256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.407225    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xjmp\" (UniqueName: \"kubernetes.io/projected/488f3620-cb7b-4a01-9520-4f8ec4e4a605-kube-api-access-6xjmp\") pod \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") "
Sep 15 11:51:04.407414 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.407342    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488f3620-cb7b-4a01-9520-4f8ec4e4a605-plugins-config-volume\") pod \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\" (UID: \"488f3620-cb7b-4a01-9520-4f8ec4e4a605\") "
Sep 15 11:51:04.407722 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.407693    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488f3620-cb7b-4a01-9520-4f8ec4e4a605-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "488f3620-cb7b-4a01-9520-4f8ec4e4a605" (UID: "488f3620-cb7b-4a01-9520-4f8ec4e4a605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:04.409451 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.409426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488f3620-cb7b-4a01-9520-4f8ec4e4a605-kube-api-access-6xjmp" (OuterVolumeSpecName: "kube-api-access-6xjmp") pod "488f3620-cb7b-4a01-9520-4f8ec4e4a605" (UID: "488f3620-cb7b-4a01-9520-4f8ec4e4a605"). InnerVolumeSpecName "kube-api-access-6xjmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:04.509241 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.509202    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/488f3620-cb7b-4a01-9520-4f8ec4e4a605-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:04.509241 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.509234    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xjmp\" (UniqueName: \"kubernetes.io/projected/488f3620-cb7b-4a01-9520-4f8ec4e4a605-kube-api-access-6xjmp\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:04.666228 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.666193    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9c7b6a9-76b0-494c-b4db-62d0de2e9325" path="/var/lib/kubelet/pods/f9c7b6a9-76b0-494c-b4db-62d0de2e9325/volumes"
Sep 15 11:51:04.737099 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.737065    2574 generic.go:356] "Generic (PLEG): container finished" podID="488f3620-cb7b-4a01-9520-4f8ec4e4a605" containerID="36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d" exitCode=0
Sep 15 11:51:04.737261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.737106    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"
Sep 15 11:51:04.737261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.737147    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b" event={"ID":"488f3620-cb7b-4a01-9520-4f8ec4e4a605","Type":"ContainerDied","Data":"36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d"}
Sep 15 11:51:04.737261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.737197    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b" event={"ID":"488f3620-cb7b-4a01-9520-4f8ec4e4a605","Type":"ContainerDied","Data":"cc84302071049067c6ccb7f2c26d70e6f1bb2f9e7c84db44bae2fa17c455646b"}
Sep 15 11:51:04.737261 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.737218    2574 scope.go:117] "RemoveContainer" containerID="36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d"
Sep 15 11:51:04.746345 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.746323    2574 scope.go:117] "RemoveContainer" containerID="36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d"
Sep 15 11:51:04.746643 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:51:04.746621    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d\": container with ID starting with 36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d not found: ID does not exist" containerID="36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d"
Sep 15 11:51:04.746697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.746650    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d"} err="failed to get container status \"36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d\": rpc error: code = NotFound desc = could not find container \"36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d\": container with ID starting with 36ae08d5bafbe64a875dfa4a01bea0260dbeeaab9f42426202c5aadd1bc1937d not found: ID does not exist"
Sep 15 11:51:04.755340 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.755317    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"]
Sep 15 11:51:04.761638 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:04.761617    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-01df46-64ff4f954f-cfj7b"]
Sep 15 11:51:06.672542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:06.672483    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488f3620-cb7b-4a01-9520-4f8ec4e4a605" path="/var/lib/kubelet/pods/488f3620-cb7b-4a01-9520-4f8ec4e4a605/volumes"
Sep 15 11:51:06.851137 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:06.851108    2574 scope.go:117] "RemoveContainer" containerID="4370617d23185c4c14825edc0552e5942dd07c0b4c5d14c0bbf5843917afbfc0"
Sep 15 11:51:08.711285 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711247    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"]
Sep 15 11:51:08.711744 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711729    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9c7b6a9-76b0-494c-b4db-62d0de2e9325" containerName="revoke-keys"
Sep 15 11:51:08.711794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711746    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9c7b6a9-76b0-494c-b4db-62d0de2e9325" containerName="revoke-keys"
Sep 15 11:51:08.711794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711767    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488f3620-cb7b-4a01-9520-4f8ec4e4a605" containerName="payload-processing"
Sep 15 11:51:08.711794 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711774    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="488f3620-cb7b-4a01-9520-4f8ec4e4a605" containerName="payload-processing"
Sep 15 11:51:08.711888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711842    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="488f3620-cb7b-4a01-9520-4f8ec4e4a605" containerName="payload-processing"
Sep 15 11:51:08.711888 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.711856    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9c7b6a9-76b0-494c-b4db-62d0de2e9325" containerName="revoke-keys"
Sep 15 11:51:08.715066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.715045    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:08.717190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.717161    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:51:08.717649 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.717632    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:51:08.723991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.723970    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"]
Sep 15 11:51:08.754727 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.754694    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e97d1d-7602-4901-b482-8a8cce05b3ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth\" (UID: \"46e97d1d-7602-4901-b482-8a8cce05b3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:08.856268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.856228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e97d1d-7602-4901-b482-8a8cce05b3ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth\" (UID: \"46e97d1d-7602-4901-b482-8a8cce05b3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:08.856929 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:08.856909    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e97d1d-7602-4901-b482-8a8cce05b3ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth\" (UID: \"46e97d1d-7602-4901-b482-8a8cce05b3ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:09.027814 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:09.027727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:09.317072 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:09.317046    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"]
Sep 15 11:51:09.318825 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:51:09.318797    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e97d1d_7602_4901_b482_8a8cce05b3ac.slice/crio-58a94f3e2bb6f3e391ac7ef0ef32c5b2ed50b4d3c3b02eab2eb8d9773eb546fe WatchSource:0}: Error finding container 58a94f3e2bb6f3e391ac7ef0ef32c5b2ed50b4d3c3b02eab2eb8d9773eb546fe: Status 404 returned error can't find the container with id 58a94f3e2bb6f3e391ac7ef0ef32c5b2ed50b4d3c3b02eab2eb8d9773eb546fe
Sep 15 11:51:09.768034 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:09.767993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth" event={"ID":"46e97d1d-7602-4901-b482-8a8cce05b3ac","Type":"ContainerStarted","Data":"b424a1c3f4cac85c21d5be9ca41008baa684d3380f1c17f507f77bcbdf616e38"}
Sep 15 11:51:09.768396 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:09.768054    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth" event={"ID":"46e97d1d-7602-4901-b482-8a8cce05b3ac","Type":"ContainerStarted","Data":"58a94f3e2bb6f3e391ac7ef0ef32c5b2ed50b4d3c3b02eab2eb8d9773eb546fe"}
Sep 15 11:51:09.783177 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:09.783123    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth" podStartSLOduration=1.783107406 podStartE2EDuration="1.783107406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:09.781329586 +0000 UTC m=+1563.559005844" watchObservedRunningTime="2026-09-15 11:51:09.783107406 +0000 UTC m=+1563.560783662"
Sep 15 11:51:10.776318 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:10.776286    2574 generic.go:356] "Generic (PLEG): container finished" podID="46e97d1d-7602-4901-b482-8a8cce05b3ac" containerID="b424a1c3f4cac85c21d5be9ca41008baa684d3380f1c17f507f77bcbdf616e38" exitCode=0
Sep 15 11:51:10.776723 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:10.776363    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth" event={"ID":"46e97d1d-7602-4901-b482-8a8cce05b3ac","Type":"ContainerDied","Data":"b424a1c3f4cac85c21d5be9ca41008baa684d3380f1c17f507f77bcbdf616e38"}
Sep 15 11:51:11.176538 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.176486    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"]
Sep 15 11:51:11.180565 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.180543    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:11.189281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.189260    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"]
Sep 15 11:51:11.280038 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.279984    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c4f1a6-7253-47bc-9ce2-962126ca5dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7\" (UID: \"02c4f1a6-7253-47bc-9ce2-962126ca5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:11.380804 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.380772    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c4f1a6-7253-47bc-9ce2-962126ca5dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7\" (UID: \"02c4f1a6-7253-47bc-9ce2-962126ca5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:11.381389 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.381367    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c4f1a6-7253-47bc-9ce2-962126ca5dba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7\" (UID: \"02c4f1a6-7253-47bc-9ce2-962126ca5dba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:11.493979 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.493897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:11.769896 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.769864    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"]
Sep 15 11:51:11.771326 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:51:11.771297    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c4f1a6_7253_47bc_9ce2_962126ca5dba.slice/crio-c3bd6908c3c91f18c8c587986fc9201e1d28c5a4d81da5c66443cb567df5f404 WatchSource:0}: Error finding container c3bd6908c3c91f18c8c587986fc9201e1d28c5a4d81da5c66443cb567df5f404: Status 404 returned error can't find the container with id c3bd6908c3c91f18c8c587986fc9201e1d28c5a4d81da5c66443cb567df5f404
Sep 15 11:51:11.783658 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.783627    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7" event={"ID":"02c4f1a6-7253-47bc-9ce2-962126ca5dba","Type":"ContainerStarted","Data":"c3bd6908c3c91f18c8c587986fc9201e1d28c5a4d81da5c66443cb567df5f404"}
Sep 15 11:51:11.916357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.916328    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:11.986974 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.986946    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e97d1d-7602-4901-b482-8a8cce05b3ac-maas-api-service-ca\") pod \"46e97d1d-7602-4901-b482-8a8cce05b3ac\" (UID: \"46e97d1d-7602-4901-b482-8a8cce05b3ac\") "
Sep 15 11:51:11.987302 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:11.987280    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e97d1d-7602-4901-b482-8a8cce05b3ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46e97d1d-7602-4901-b482-8a8cce05b3ac" (UID: "46e97d1d-7602-4901-b482-8a8cce05b3ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:12.088264 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:12.088229    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46e97d1d-7602-4901-b482-8a8cce05b3ac-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:12.790839 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:12.790761    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"
Sep 15 11:51:12.790839 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:12.790820    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth" event={"ID":"46e97d1d-7602-4901-b482-8a8cce05b3ac","Type":"ContainerDied","Data":"58a94f3e2bb6f3e391ac7ef0ef32c5b2ed50b4d3c3b02eab2eb8d9773eb546fe"}
Sep 15 11:51:12.791239 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:12.790879    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a94f3e2bb6f3e391ac7ef0ef32c5b2ed50b4d3c3b02eab2eb8d9773eb546fe"
Sep 15 11:51:12.792728 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:12.792678    2574 generic.go:356] "Generic (PLEG): container finished" podID="02c4f1a6-7253-47bc-9ce2-962126ca5dba" containerID="5d83eed308364339f251403e27f0d9021fef0891ccdcbd02394e94db6a0c7ddc" exitCode=0
Sep 15 11:51:12.792728 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:12.792726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7" event={"ID":"02c4f1a6-7253-47bc-9ce2-962126ca5dba","Type":"ContainerDied","Data":"5d83eed308364339f251403e27f0d9021fef0891ccdcbd02394e94db6a0c7ddc"}
Sep 15 11:51:13.600187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:13.600153    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"]
Sep 15 11:51:13.622224 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:13.622185    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"]
Sep 15 11:51:13.937406 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:13.937379    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:14.005368 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:14.005330    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c4f1a6-7253-47bc-9ce2-962126ca5dba-maas-api-service-ca\") pod \"02c4f1a6-7253-47bc-9ce2-962126ca5dba\" (UID: \"02c4f1a6-7253-47bc-9ce2-962126ca5dba\") "
Sep 15 11:51:14.005701 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:14.005677    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c4f1a6-7253-47bc-9ce2-962126ca5dba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02c4f1a6-7253-47bc-9ce2-962126ca5dba" (UID: "02c4f1a6-7253-47bc-9ce2-962126ca5dba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:14.005823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:14.005810    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c4f1a6-7253-47bc-9ce2-962126ca5dba-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:14.805583 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:14.805551    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"
Sep 15 11:51:14.805785 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:14.805601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7" event={"ID":"02c4f1a6-7253-47bc-9ce2-962126ca5dba","Type":"ContainerDied","Data":"c3bd6908c3c91f18c8c587986fc9201e1d28c5a4d81da5c66443cb567df5f404"}
Sep 15 11:51:14.805785 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:14.805673    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3bd6908c3c91f18c8c587986fc9201e1d28c5a4d81da5c66443cb567df5f404"
Sep 15 11:51:24.576245 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.576208    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"]
Sep 15 11:51:24.576708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.576482    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5" podUID="2c77cf0d-5c41-44f1-8442-06e5575866a3" containerName="payload-processing" containerID="cri-o://14f79880ef31c00ac37ec792921ca0601edb25e6991fd109a9dc6620daf07a7b" gracePeriod=30
Sep 15 11:51:24.584582 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.584488    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"]
Sep 15 11:51:24.585949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.584752    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt" podUID="4b2afe84-3455-4811-a54b-de889d082ee1" containerName="payload-pre-processing" containerID="cri-o://d75a29d11e549b3e06786d8467aef6149b3b3d523354604e6fa9eff4b92d566a" gracePeriod=30
Sep 15 11:51:24.755654 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.755590    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"]
Sep 15 11:51:24.758166 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.758069    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-9bd868-ztjth"]
Sep 15 11:51:24.869007 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.868975    2574 generic.go:356] "Generic (PLEG): container finished" podID="4b2afe84-3455-4811-a54b-de889d082ee1" containerID="d75a29d11e549b3e06786d8467aef6149b3b3d523354604e6fa9eff4b92d566a" exitCode=0
Sep 15 11:51:24.869122 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.869050    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt" event={"ID":"4b2afe84-3455-4811-a54b-de889d082ee1","Type":"ContainerDied","Data":"d75a29d11e549b3e06786d8467aef6149b3b3d523354604e6fa9eff4b92d566a"}
Sep 15 11:51:24.869122 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.869112    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt" event={"ID":"4b2afe84-3455-4811-a54b-de889d082ee1","Type":"ContainerDied","Data":"d16de5990fd1bb6f051d27fb5fe74609e996f5bec5c45120d6b6ad8db789cc17"}
Sep 15 11:51:24.869232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.869137    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d16de5990fd1bb6f051d27fb5fe74609e996f5bec5c45120d6b6ad8db789cc17"
Sep 15 11:51:24.870447 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.870426    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c77cf0d-5c41-44f1-8442-06e5575866a3" containerID="14f79880ef31c00ac37ec792921ca0601edb25e6991fd109a9dc6620daf07a7b" exitCode=0
Sep 15 11:51:24.870565 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.870511    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5" event={"ID":"2c77cf0d-5c41-44f1-8442-06e5575866a3","Type":"ContainerDied","Data":"14f79880ef31c00ac37ec792921ca0601edb25e6991fd109a9dc6620daf07a7b"}
Sep 15 11:51:24.870565 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.870551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5" event={"ID":"2c77cf0d-5c41-44f1-8442-06e5575866a3","Type":"ContainerDied","Data":"9cebd3022a83d26bf31bd215b0cd8911ea22c1e11b1f73c32b1069bf6c8d18fa"}
Sep 15 11:51:24.870682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.870567    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cebd3022a83d26bf31bd215b0cd8911ea22c1e11b1f73c32b1069bf6c8d18fa"
Sep 15 11:51:24.874135 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.874116    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:51:24.877255 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:24.877239    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:51:25.018703 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.018676    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c77cf0d-5c41-44f1-8442-06e5575866a3-plugins-config-volume\") pod \"2c77cf0d-5c41-44f1-8442-06e5575866a3\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") "
Sep 15 11:51:25.018883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.018747    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn2f5\" (UniqueName: \"kubernetes.io/projected/2c77cf0d-5c41-44f1-8442-06e5575866a3-kube-api-access-bn2f5\") pod \"2c77cf0d-5c41-44f1-8442-06e5575866a3\" (UID: \"2c77cf0d-5c41-44f1-8442-06e5575866a3\") "
Sep 15 11:51:25.018883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.018783    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2afe84-3455-4811-a54b-de889d082ee1-plugins-config-volume\") pod \"4b2afe84-3455-4811-a54b-de889d082ee1\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") "
Sep 15 11:51:25.018883 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.018823    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j974z\" (UniqueName: \"kubernetes.io/projected/4b2afe84-3455-4811-a54b-de889d082ee1-kube-api-access-j974z\") pod \"4b2afe84-3455-4811-a54b-de889d082ee1\" (UID: \"4b2afe84-3455-4811-a54b-de889d082ee1\") "
Sep 15 11:51:25.019056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.018997    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c77cf0d-5c41-44f1-8442-06e5575866a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c77cf0d-5c41-44f1-8442-06e5575866a3" (UID: "2c77cf0d-5c41-44f1-8442-06e5575866a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:25.019160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.019135    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b2afe84-3455-4811-a54b-de889d082ee1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b2afe84-3455-4811-a54b-de889d082ee1" (UID: "4b2afe84-3455-4811-a54b-de889d082ee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:25.019293 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.019201    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c77cf0d-5c41-44f1-8442-06e5575866a3-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:25.020814 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.020791    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b2afe84-3455-4811-a54b-de889d082ee1-kube-api-access-j974z" (OuterVolumeSpecName: "kube-api-access-j974z") pod "4b2afe84-3455-4811-a54b-de889d082ee1" (UID: "4b2afe84-3455-4811-a54b-de889d082ee1"). InnerVolumeSpecName "kube-api-access-j974z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:25.020910 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.020841    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c77cf0d-5c41-44f1-8442-06e5575866a3-kube-api-access-bn2f5" (OuterVolumeSpecName: "kube-api-access-bn2f5") pod "2c77cf0d-5c41-44f1-8442-06e5575866a3" (UID: "2c77cf0d-5c41-44f1-8442-06e5575866a3"). InnerVolumeSpecName "kube-api-access-bn2f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:25.120422 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.120322    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b2afe84-3455-4811-a54b-de889d082ee1-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:25.120422 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.120357    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j974z\" (UniqueName: \"kubernetes.io/projected/4b2afe84-3455-4811-a54b-de889d082ee1-kube-api-access-j974z\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:25.120422 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.120371    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2f5\" (UniqueName: \"kubernetes.io/projected/2c77cf0d-5c41-44f1-8442-06e5575866a3-kube-api-access-bn2f5\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:25.876437 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.876355    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"
Sep 15 11:51:25.876923 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.876355    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"
Sep 15 11:51:25.902647 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.902619    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"]
Sep 15 11:51:25.909313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.909293    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-9bd868-549b8c464f-5v9f5"]
Sep 15 11:51:25.918375 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.918353    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"]
Sep 15 11:51:25.923860 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:25.923833    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-9bd868-5c9d559666-rdllt"]
Sep 15 11:51:26.666130 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:26.666094    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c77cf0d-5c41-44f1-8442-06e5575866a3" path="/var/lib/kubelet/pods/2c77cf0d-5c41-44f1-8442-06e5575866a3/volumes"
Sep 15 11:51:26.666451 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:26.666438    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e97d1d-7602-4901-b482-8a8cce05b3ac" path="/var/lib/kubelet/pods/46e97d1d-7602-4901-b482-8a8cce05b3ac/volumes"
Sep 15 11:51:26.666781 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:26.666769    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b2afe84-3455-4811-a54b-de889d082ee1" path="/var/lib/kubelet/pods/4b2afe84-3455-4811-a54b-de889d082ee1/volumes"
Sep 15 11:51:28.157576 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.157530    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"]
Sep 15 11:51:28.160666 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.160637    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3e7da-5lcn7"]
Sep 15 11:51:28.620313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.620200    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn" podUID="ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" containerName="main" containerID="cri-o://660619d1e5532e6407bf3ebaf9de94977d74b81c05926eb36e7f85afed6e2497" gracePeriod=45
Sep 15 11:51:28.642946 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.642903    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9" podUID="2b5d9369-d996-4df8-b073-5ae2a20cc3e4" containerName="main" containerID="cri-o://363df011078b16b2ca58afb361f5bc6727f11d92dfb15dcb8a0433f480763e9f" gracePeriod=45
Sep 15 11:51:28.665296 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.665258    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c4f1a6-7253-47bc-9ce2-962126ca5dba" path="/var/lib/kubelet/pods/02c4f1a6-7253-47bc-9ce2-962126ca5dba/volumes"
Sep 15 11:51:28.898895 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.898861    2574 generic.go:356] "Generic (PLEG): container finished" podID="ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" containerID="660619d1e5532e6407bf3ebaf9de94977d74b81c05926eb36e7f85afed6e2497" exitCode=0
Sep 15 11:51:28.899073 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.898936    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn" event={"ID":"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f","Type":"ContainerDied","Data":"660619d1e5532e6407bf3ebaf9de94977d74b81c05926eb36e7f85afed6e2497"}
Sep 15 11:51:28.899073 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.898975    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn" event={"ID":"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f","Type":"ContainerDied","Data":"8507711f3e741a8ad26b020da80103113ba5134bf82a354df6a98a49da6c5fe1"}
Sep 15 11:51:28.899073 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.898987    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8507711f3e741a8ad26b020da80103113ba5134bf82a354df6a98a49da6c5fe1"
Sep 15 11:51:28.899259 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.899234    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:51:28.901060 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.901030    2574 generic.go:356] "Generic (PLEG): container finished" podID="2b5d9369-d996-4df8-b073-5ae2a20cc3e4" containerID="363df011078b16b2ca58afb361f5bc6727f11d92dfb15dcb8a0433f480763e9f" exitCode=0
Sep 15 11:51:28.901174 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.901077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9" event={"ID":"2b5d9369-d996-4df8-b073-5ae2a20cc3e4","Type":"ContainerDied","Data":"363df011078b16b2ca58afb361f5bc6727f11d92dfb15dcb8a0433f480763e9f"}
Sep 15 11:51:28.929311 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.929287    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:51:28.960900 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.960871    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tmp-dir\") pod \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") "
Sep 15 11:51:28.961093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.960920    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tls-certs\") pod \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") "
Sep 15 11:51:28.961093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.960959    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-home\") pod \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") "
Sep 15 11:51:28.961093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.960986    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznnf\" (UniqueName: \"kubernetes.io/projected/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-kube-api-access-hznnf\") pod \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") "
Sep 15 11:51:28.961093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961023    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-home\") pod \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") "
Sep 15 11:51:28.961093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961057    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tls-certs\") pod \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") "
Sep 15 11:51:28.961357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-dshm\") pod \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") "
Sep 15 11:51:28.961357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961126    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-model-cache\") pod \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") "
Sep 15 11:51:28.961357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961165    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" (UID: "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.961357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961183    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tmp-dir\") pod \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\" (UID: \"2b5d9369-d996-4df8-b073-5ae2a20cc3e4\") "
Sep 15 11:51:28.961357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961279    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-dshm\") pod \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") "
Sep 15 11:51:28.961357 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961349    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-model-cache\") pod \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") "
Sep 15 11:51:28.961730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961375    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b5d9369-d996-4df8-b073-5ae2a20cc3e4" (UID: "2b5d9369-d996-4df8-b073-5ae2a20cc3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.961730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pth9m\" (UniqueName: \"kubernetes.io/projected/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-kube-api-access-pth9m\") pod \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\" (UID: \"ab82c0c1-eb94-44e9-967f-e1e0ab56f22f\") "
Sep 15 11:51:28.961730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961608    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b5d9369-d996-4df8-b073-5ae2a20cc3e4" (UID: "2b5d9369-d996-4df8-b073-5ae2a20cc3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.962184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961875    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-home" (OuterVolumeSpecName: "home") pod "2b5d9369-d996-4df8-b073-5ae2a20cc3e4" (UID: "2b5d9369-d996-4df8-b073-5ae2a20cc3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.962184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961921    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:28.962184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961942    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:28.962184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.961957    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:28.962184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.962143    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-home" (OuterVolumeSpecName: "home") pod "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" (UID: "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.962184 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.962164    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" (UID: "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.964313 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.964290    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-kube-api-access-pth9m" (OuterVolumeSpecName: "kube-api-access-pth9m") pod "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" (UID: "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f"). InnerVolumeSpecName "kube-api-access-pth9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:28.964440 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.964389    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b5d9369-d996-4df8-b073-5ae2a20cc3e4" (UID: "2b5d9369-d996-4df8-b073-5ae2a20cc3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:28.964674 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.964644    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-dshm" (OuterVolumeSpecName: "dshm") pod "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" (UID: "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.965013 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.964985    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-dshm" (OuterVolumeSpecName: "dshm") pod "2b5d9369-d996-4df8-b073-5ae2a20cc3e4" (UID: "2b5d9369-d996-4df8-b073-5ae2a20cc3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:51:28.965013 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.964993    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-kube-api-access-hznnf" (OuterVolumeSpecName: "kube-api-access-hznnf") pod "2b5d9369-d996-4df8-b073-5ae2a20cc3e4" (UID: "2b5d9369-d996-4df8-b073-5ae2a20cc3e4"). InnerVolumeSpecName "kube-api-access-hznnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:28.965762 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:28.965744    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" (UID: "ab82c0c1-eb94-44e9-967f-e1e0ab56f22f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:51:29.062949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062916    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.062949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062942    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.062949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062951    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznnf\" (UniqueName: \"kubernetes.io/projected/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-kube-api-access-hznnf\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.062949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062959    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.063202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062968    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.063202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062976    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b5d9369-d996-4df8-b073-5ae2a20cc3e4-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.063202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062983    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.063202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062990    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.063202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.062999    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pth9m\" (UniqueName: \"kubernetes.io/projected/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f-kube-api-access-pth9m\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:29.909396 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.909368    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"
Sep 15 11:51:29.909396 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.909381    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"
Sep 15 11:51:29.909897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.909398    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9" event={"ID":"2b5d9369-d996-4df8-b073-5ae2a20cc3e4","Type":"ContainerDied","Data":"565d58c5fd3c350dde9b524ac8a0bcac85dd35151d380a0b1dbab61ae9b93e1a"}
Sep 15 11:51:29.909897 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.909445    2574 scope.go:117] "RemoveContainer" containerID="363df011078b16b2ca58afb361f5bc6727f11d92dfb15dcb8a0433f480763e9f"
Sep 15 11:51:29.934000 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.933973    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"]
Sep 15 11:51:29.940147 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.940125    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/simulator-w0-9bd868-kserve-757ff85c9d-jwpm9"]
Sep 15 11:51:29.951898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.951876    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"]
Sep 15 11:51:29.958314 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:29.958293    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9bd868/embedding-w0-9bd868-kserve-76f799d4f9-xg2qn"]
Sep 15 11:51:30.665117 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:30.665083    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b5d9369-d996-4df8-b073-5ae2a20cc3e4" path="/var/lib/kubelet/pods/2b5d9369-d996-4df8-b073-5ae2a20cc3e4/volumes"
Sep 15 11:51:30.665545 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:30.665531    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" path="/var/lib/kubelet/pods/ab82c0c1-eb94-44e9-967f-e1e0ab56f22f/volumes"
Sep 15 11:51:35.703421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703388    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"]
Sep 15 11:51:35.703867 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703853    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c4f1a6-7253-47bc-9ce2-962126ca5dba" containerName="revoke-keys"
Sep 15 11:51:35.703912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703869    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c4f1a6-7253-47bc-9ce2-962126ca5dba" containerName="revoke-keys"
Sep 15 11:51:35.703912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703879    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" containerName="main"
Sep 15 11:51:35.703912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703886    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" containerName="main"
Sep 15 11:51:35.703912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703901    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c77cf0d-5c41-44f1-8442-06e5575866a3" containerName="payload-processing"
Sep 15 11:51:35.703912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703907    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c77cf0d-5c41-44f1-8442-06e5575866a3" containerName="payload-processing"
Sep 15 11:51:35.703912 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703914    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e97d1d-7602-4901-b482-8a8cce05b3ac" containerName="revoke-keys"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703919    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e97d1d-7602-4901-b482-8a8cce05b3ac" containerName="revoke-keys"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703927    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b2afe84-3455-4811-a54b-de889d082ee1" containerName="payload-pre-processing"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703932    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b2afe84-3455-4811-a54b-de889d082ee1" containerName="payload-pre-processing"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703938    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b5d9369-d996-4df8-b073-5ae2a20cc3e4" containerName="main"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.703943    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b5d9369-d996-4df8-b073-5ae2a20cc3e4" containerName="main"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.704028    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b2afe84-3455-4811-a54b-de889d082ee1" containerName="payload-pre-processing"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.704040    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b5d9369-d996-4df8-b073-5ae2a20cc3e4" containerName="main"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.704046    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab82c0c1-eb94-44e9-967f-e1e0ab56f22f" containerName="main"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.704052    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c4f1a6-7253-47bc-9ce2-962126ca5dba" containerName="revoke-keys"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.704059    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e97d1d-7602-4901-b482-8a8cce05b3ac" containerName="revoke-keys"
Sep 15 11:51:35.704108 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.704066    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c77cf0d-5c41-44f1-8442-06e5575866a3" containerName="payload-processing"
Sep 15 11:51:35.708926 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.708906    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:35.711177 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.711154    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:51:35.711535 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.711484    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:51:35.714096 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.714069    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"]
Sep 15 11:51:35.832582 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.832553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aad8ce-adf9-4af6-bdf2-a461d4136ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz\" (UID: \"82aad8ce-adf9-4af6-bdf2-a461d4136ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:35.933632 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.933601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aad8ce-adf9-4af6-bdf2-a461d4136ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz\" (UID: \"82aad8ce-adf9-4af6-bdf2-a461d4136ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:35.934235 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:35.934214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aad8ce-adf9-4af6-bdf2-a461d4136ca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz\" (UID: \"82aad8ce-adf9-4af6-bdf2-a461d4136ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:36.021744 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:36.021668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:36.292253 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:36.292227    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"]
Sep 15 11:51:36.293873 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:51:36.293846    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82aad8ce_adf9_4af6_bdf2_a461d4136ca9.slice/crio-317a27fbf6904ac8e593b2c9f77ffbc0218a3cb26fdeec978afe9250be82f86b WatchSource:0}: Error finding container 317a27fbf6904ac8e593b2c9f77ffbc0218a3cb26fdeec978afe9250be82f86b: Status 404 returned error can't find the container with id 317a27fbf6904ac8e593b2c9f77ffbc0218a3cb26fdeec978afe9250be82f86b
Sep 15 11:51:36.953355 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:36.953322    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz" event={"ID":"82aad8ce-adf9-4af6-bdf2-a461d4136ca9","Type":"ContainerStarted","Data":"5d800d418556881a78e2ee6ea548ed34b3501671a1754172e3e4dce3bc26d00a"}
Sep 15 11:51:36.953757 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:36.953374    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz" event={"ID":"82aad8ce-adf9-4af6-bdf2-a461d4136ca9","Type":"ContainerStarted","Data":"317a27fbf6904ac8e593b2c9f77ffbc0218a3cb26fdeec978afe9250be82f86b"}
Sep 15 11:51:36.970358 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:36.970313    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz" podStartSLOduration=1.970297821 podStartE2EDuration="1.970297821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:36.967594241 +0000 UTC m=+1590.745270497" watchObservedRunningTime="2026-09-15 11:51:36.970297821 +0000 UTC m=+1590.747974078"
Sep 15 11:51:37.969377 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:37.969337    2574 generic.go:356] "Generic (PLEG): container finished" podID="82aad8ce-adf9-4af6-bdf2-a461d4136ca9" containerID="5d800d418556881a78e2ee6ea548ed34b3501671a1754172e3e4dce3bc26d00a" exitCode=0
Sep 15 11:51:37.969891 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:37.969408    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz" event={"ID":"82aad8ce-adf9-4af6-bdf2-a461d4136ca9","Type":"ContainerDied","Data":"5d800d418556881a78e2ee6ea548ed34b3501671a1754172e3e4dce3bc26d00a"}
Sep 15 11:51:39.115743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.115717    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:39.164741 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.164710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aad8ce-adf9-4af6-bdf2-a461d4136ca9-maas-api-service-ca\") pod \"82aad8ce-adf9-4af6-bdf2-a461d4136ca9\" (UID: \"82aad8ce-adf9-4af6-bdf2-a461d4136ca9\") "
Sep 15 11:51:39.165061 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.165041    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82aad8ce-adf9-4af6-bdf2-a461d4136ca9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "82aad8ce-adf9-4af6-bdf2-a461d4136ca9" (UID: "82aad8ce-adf9-4af6-bdf2-a461d4136ca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:39.265618 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.265536    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82aad8ce-adf9-4af6-bdf2-a461d4136ca9-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:39.980893 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.980870    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"
Sep 15 11:51:39.981057 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.980866    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz" event={"ID":"82aad8ce-adf9-4af6-bdf2-a461d4136ca9","Type":"ContainerDied","Data":"317a27fbf6904ac8e593b2c9f77ffbc0218a3cb26fdeec978afe9250be82f86b"}
Sep 15 11:51:39.981057 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:39.980961    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317a27fbf6904ac8e593b2c9f77ffbc0218a3cb26fdeec978afe9250be82f86b"
Sep 15 11:51:46.382145 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.382098    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"]
Sep 15 11:51:46.384168 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.384147    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-01611d-29824545-r8gz2"]
Sep 15 11:51:46.413018 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.412986    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"]
Sep 15 11:51:46.413294 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.413264    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng" podUID="6bb6ad2b-2133-44f7-8874-528ba6354574" containerName="payload-processing" containerID="cri-o://82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753" gracePeriod=30
Sep 15 11:51:46.645238 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.645207    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"]
Sep 15 11:51:46.649061 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.649037    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-01611d-8fvvz"]
Sep 15 11:51:46.666936 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.666816    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82aad8ce-adf9-4af6-bdf2-a461d4136ca9" path="/var/lib/kubelet/pods/82aad8ce-adf9-4af6-bdf2-a461d4136ca9/volumes"
Sep 15 11:51:46.667134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.667120    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b497d16-5273-46cc-a072-03a412e422ff" path="/var/lib/kubelet/pods/9b497d16-5273-46cc-a072-03a412e422ff/volumes"
Sep 15 11:51:46.683337 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.683315    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:51:46.841272 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.841239    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdf6l\" (UniqueName: \"kubernetes.io/projected/6bb6ad2b-2133-44f7-8874-528ba6354574-kube-api-access-gdf6l\") pod \"6bb6ad2b-2133-44f7-8874-528ba6354574\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") "
Sep 15 11:51:46.841272 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.841273    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb6ad2b-2133-44f7-8874-528ba6354574-plugins-config-volume\") pod \"6bb6ad2b-2133-44f7-8874-528ba6354574\" (UID: \"6bb6ad2b-2133-44f7-8874-528ba6354574\") "
Sep 15 11:51:46.841682 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.841658    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb6ad2b-2133-44f7-8874-528ba6354574-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6bb6ad2b-2133-44f7-8874-528ba6354574" (UID: "6bb6ad2b-2133-44f7-8874-528ba6354574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:46.843344 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.843321    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb6ad2b-2133-44f7-8874-528ba6354574-kube-api-access-gdf6l" (OuterVolumeSpecName: "kube-api-access-gdf6l") pod "6bb6ad2b-2133-44f7-8874-528ba6354574" (UID: "6bb6ad2b-2133-44f7-8874-528ba6354574"). InnerVolumeSpecName "kube-api-access-gdf6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:51:46.942644 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.942569    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdf6l\" (UniqueName: \"kubernetes.io/projected/6bb6ad2b-2133-44f7-8874-528ba6354574-kube-api-access-gdf6l\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:46.942644 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:46.942591    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb6ad2b-2133-44f7-8874-528ba6354574-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:47.020902 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.020869    2574 generic.go:356] "Generic (PLEG): container finished" podID="6bb6ad2b-2133-44f7-8874-528ba6354574" containerID="82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753" exitCode=0
Sep 15 11:51:47.021083 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.020910    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"
Sep 15 11:51:47.021083 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.020947    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng" event={"ID":"6bb6ad2b-2133-44f7-8874-528ba6354574","Type":"ContainerDied","Data":"82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753"}
Sep 15 11:51:47.021083 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.020998    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng" event={"ID":"6bb6ad2b-2133-44f7-8874-528ba6354574","Type":"ContainerDied","Data":"5fc47ee42e539ffa0c23eb13b9af202d37d0b95f2137bd8f2cbe19fea71b7c6c"}
Sep 15 11:51:47.021083 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.021021    2574 scope.go:117] "RemoveContainer" containerID="82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753"
Sep 15 11:51:47.030489 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.030471    2574 scope.go:117] "RemoveContainer" containerID="82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753"
Sep 15 11:51:47.030755 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:51:47.030735    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753\": container with ID starting with 82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753 not found: ID does not exist" containerID="82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753"
Sep 15 11:51:47.030800 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.030763    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753"} err="failed to get container status \"82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753\": rpc error: code = NotFound desc = could not find container \"82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753\": container with ID starting with 82f21e0c170c25412d9e27ad84bbad661ff84ffcbd9b62ded0a5ea90506b3753 not found: ID does not exist"
Sep 15 11:51:47.043262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.043236    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"]
Sep 15 11:51:47.050989 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:47.050969    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-01611d-579957d8c5-7p9ng"]
Sep 15 11:51:48.665608 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:48.665576    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb6ad2b-2133-44f7-8874-528ba6354574" path="/var/lib/kubelet/pods/6bb6ad2b-2133-44f7-8874-528ba6354574/volumes"
Sep 15 11:51:53.613279 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613243    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"]
Sep 15 11:51:53.613823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613803    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82aad8ce-adf9-4af6-bdf2-a461d4136ca9" containerName="revoke-keys"
Sep 15 11:51:53.613875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613827    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="82aad8ce-adf9-4af6-bdf2-a461d4136ca9" containerName="revoke-keys"
Sep 15 11:51:53.613875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613843    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb6ad2b-2133-44f7-8874-528ba6354574" containerName="payload-processing"
Sep 15 11:51:53.613875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613851    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb6ad2b-2133-44f7-8874-528ba6354574" containerName="payload-processing"
Sep 15 11:51:53.613984 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613958    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb6ad2b-2133-44f7-8874-528ba6354574" containerName="payload-processing"
Sep 15 11:51:53.613984 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.613969    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="82aad8ce-adf9-4af6-bdf2-a461d4136ca9" containerName="revoke-keys"
Sep 15 11:51:53.618700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.618682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:53.621178 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.621147    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:51:53.621302 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.621180    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:51:53.623545 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.623516    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"]
Sep 15 11:51:53.710958 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.710922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98387ac-9d4e-4e11-9812-0513799577b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h\" (UID: \"d98387ac-9d4e-4e11-9812-0513799577b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:53.812631 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.812597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98387ac-9d4e-4e11-9812-0513799577b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h\" (UID: \"d98387ac-9d4e-4e11-9812-0513799577b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:53.813232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.813214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98387ac-9d4e-4e11-9812-0513799577b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h\" (UID: \"d98387ac-9d4e-4e11-9812-0513799577b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:53.930881 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:53.930845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:54.210710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:54.209822    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"]
Sep 15 11:51:54.214337 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:51:54.214302    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98387ac_9d4e_4e11_9812_0513799577b2.slice/crio-a5e9b8274b23b02217dc268c15610aab852b4f368b18b793b4cffa66380762ba WatchSource:0}: Error finding container a5e9b8274b23b02217dc268c15610aab852b4f368b18b793b4cffa66380762ba: Status 404 returned error can't find the container with id a5e9b8274b23b02217dc268c15610aab852b4f368b18b793b4cffa66380762ba
Sep 15 11:51:55.076955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:55.076916    2574 generic.go:356] "Generic (PLEG): container finished" podID="d98387ac-9d4e-4e11-9812-0513799577b2" containerID="98b24165df50f4ccc1f63e18304923a312f36cad9d6cb26133e9684b676ce219" exitCode=0
Sep 15 11:51:55.077381 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:55.077003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h" event={"ID":"d98387ac-9d4e-4e11-9812-0513799577b2","Type":"ContainerDied","Data":"98b24165df50f4ccc1f63e18304923a312f36cad9d6cb26133e9684b676ce219"}
Sep 15 11:51:55.077381 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:55.077091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h" event={"ID":"d98387ac-9d4e-4e11-9812-0513799577b2","Type":"ContainerStarted","Data":"a5e9b8274b23b02217dc268c15610aab852b4f368b18b793b4cffa66380762ba"}
Sep 15 11:51:56.222873 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:56.222845    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:56.338223 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:56.338136    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98387ac-9d4e-4e11-9812-0513799577b2-maas-api-service-ca\") pod \"d98387ac-9d4e-4e11-9812-0513799577b2\" (UID: \"d98387ac-9d4e-4e11-9812-0513799577b2\") "
Sep 15 11:51:56.338515 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:56.338476    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98387ac-9d4e-4e11-9812-0513799577b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d98387ac-9d4e-4e11-9812-0513799577b2" (UID: "d98387ac-9d4e-4e11-9812-0513799577b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:51:56.439297 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:56.439261    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d98387ac-9d4e-4e11-9812-0513799577b2-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:51:57.089678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.089599    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"
Sep 15 11:51:57.089678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.089628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h" event={"ID":"d98387ac-9d4e-4e11-9812-0513799577b2","Type":"ContainerDied","Data":"a5e9b8274b23b02217dc268c15610aab852b4f368b18b793b4cffa66380762ba"}
Sep 15 11:51:57.089678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.089672    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e9b8274b23b02217dc268c15610aab852b4f368b18b793b4cffa66380762ba"
Sep 15 11:51:57.244965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.244933    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"]
Sep 15 11:51:57.245570 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.245552    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98387ac-9d4e-4e11-9812-0513799577b2" containerName="revoke-keys"
Sep 15 11:51:57.245673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.245573    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98387ac-9d4e-4e11-9812-0513799577b2" containerName="revoke-keys"
Sep 15 11:51:57.245734 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.245692    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98387ac-9d4e-4e11-9812-0513799577b2" containerName="revoke-keys"
Sep 15 11:51:57.249028 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.249007    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:51:57.251037 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.251015    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:51:57.251144 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.251053    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:51:57.257994 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.256799    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"]
Sep 15 11:51:57.349384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.349288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d62a54-e294-440c-8c42-80c2421902c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc\" (UID: \"07d62a54-e294-440c-8c42-80c2421902c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:51:57.450818 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.450781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d62a54-e294-440c-8c42-80c2421902c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc\" (UID: \"07d62a54-e294-440c-8c42-80c2421902c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:51:57.451424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.451402    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d62a54-e294-440c-8c42-80c2421902c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc\" (UID: \"07d62a54-e294-440c-8c42-80c2421902c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:51:57.562379 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.562348    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:51:57.836473 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:57.836446    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"]
Sep 15 11:51:57.838332 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:51:57.838299    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d62a54_e294_440c_8c42_80c2421902c5.slice/crio-6d9e7f8bc92d0dda059d1388c565f5234c7d935272ad1d3dbc00513f38c6bbfe WatchSource:0}: Error finding container 6d9e7f8bc92d0dda059d1388c565f5234c7d935272ad1d3dbc00513f38c6bbfe: Status 404 returned error can't find the container with id 6d9e7f8bc92d0dda059d1388c565f5234c7d935272ad1d3dbc00513f38c6bbfe
Sep 15 11:51:58.097162 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:58.097122    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc" event={"ID":"07d62a54-e294-440c-8c42-80c2421902c5","Type":"ContainerStarted","Data":"cf0566917502b22a348e6d3615abe31006bc9501e42d1874535d6c6a321c4121"}
Sep 15 11:51:58.097349 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:58.097178    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc" event={"ID":"07d62a54-e294-440c-8c42-80c2421902c5","Type":"ContainerStarted","Data":"6d9e7f8bc92d0dda059d1388c565f5234c7d935272ad1d3dbc00513f38c6bbfe"}
Sep 15 11:51:58.110885 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:58.110784    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc" podStartSLOduration=1.110764102 podStartE2EDuration="1.110764102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:51:58.110462852 +0000 UTC m=+1611.888139111" watchObservedRunningTime="2026-09-15 11:51:58.110764102 +0000 UTC m=+1611.888440360"
Sep 15 11:51:59.105445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:59.105367    2574 generic.go:356] "Generic (PLEG): container finished" podID="07d62a54-e294-440c-8c42-80c2421902c5" containerID="cf0566917502b22a348e6d3615abe31006bc9501e42d1874535d6c6a321c4121" exitCode=0
Sep 15 11:51:59.105821 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:51:59.105455    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc" event={"ID":"07d62a54-e294-440c-8c42-80c2421902c5","Type":"ContainerDied","Data":"cf0566917502b22a348e6d3615abe31006bc9501e42d1874535d6c6a321c4121"}
Sep 15 11:52:00.250730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:00.250707    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:52:00.276621 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:00.276591    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d62a54-e294-440c-8c42-80c2421902c5-maas-api-service-ca\") pod \"07d62a54-e294-440c-8c42-80c2421902c5\" (UID: \"07d62a54-e294-440c-8c42-80c2421902c5\") "
Sep 15 11:52:00.276946 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:00.276923    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d62a54-e294-440c-8c42-80c2421902c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "07d62a54-e294-440c-8c42-80c2421902c5" (UID: "07d62a54-e294-440c-8c42-80c2421902c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:00.377426 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:00.377339    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d62a54-e294-440c-8c42-80c2421902c5-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:01.118476 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:01.118391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc" event={"ID":"07d62a54-e294-440c-8c42-80c2421902c5","Type":"ContainerDied","Data":"6d9e7f8bc92d0dda059d1388c565f5234c7d935272ad1d3dbc00513f38c6bbfe"}
Sep 15 11:52:01.118476 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:01.118416    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"
Sep 15 11:52:01.118476 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:01.118443    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d9e7f8bc92d0dda059d1388c565f5234c7d935272ad1d3dbc00513f38c6bbfe"
Sep 15 11:52:02.106295 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:02.106261    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"]
Sep 15 11:52:02.126148 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:02.126119    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"]
Sep 15 11:52:02.138116 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:02.138089    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"]
Sep 15 11:52:04.388587 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.388551    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"]
Sep 15 11:52:04.389024 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.388764    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb" podUID="ed713dcd-c36c-4899-a060-907aa14dffad" containerName="payload-pre-processing" containerID="cri-o://594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794" gracePeriod=30
Sep 15 11:52:04.577476 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.577449    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"]
Sep 15 11:52:04.579614 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.579587    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4dbbbf-5t82h"]
Sep 15 11:52:04.643848 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.643826    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:52:04.667094 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.667058    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98387ac-9d4e-4e11-9812-0513799577b2" path="/var/lib/kubelet/pods/d98387ac-9d4e-4e11-9812-0513799577b2/volumes"
Sep 15 11:52:04.714926 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.714894    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ed713dcd-c36c-4899-a060-907aa14dffad-kube-api-access-qngm7\") pod \"ed713dcd-c36c-4899-a060-907aa14dffad\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") "
Sep 15 11:52:04.715093 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.714961    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713dcd-c36c-4899-a060-907aa14dffad-plugins-config-volume\") pod \"ed713dcd-c36c-4899-a060-907aa14dffad\" (UID: \"ed713dcd-c36c-4899-a060-907aa14dffad\") "
Sep 15 11:52:04.715337 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.715314    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed713dcd-c36c-4899-a060-907aa14dffad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed713dcd-c36c-4899-a060-907aa14dffad" (UID: "ed713dcd-c36c-4899-a060-907aa14dffad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:52:04.717065 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.717045    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed713dcd-c36c-4899-a060-907aa14dffad-kube-api-access-qngm7" (OuterVolumeSpecName: "kube-api-access-qngm7") pod "ed713dcd-c36c-4899-a060-907aa14dffad" (UID: "ed713dcd-c36c-4899-a060-907aa14dffad"). InnerVolumeSpecName "kube-api-access-qngm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:04.816068 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.816033    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qngm7\" (UniqueName: \"kubernetes.io/projected/ed713dcd-c36c-4899-a060-907aa14dffad-kube-api-access-qngm7\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:04.816068 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:04.816063    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed713dcd-c36c-4899-a060-907aa14dffad-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:05.143892 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.143858    2574 generic.go:356] "Generic (PLEG): container finished" podID="ed713dcd-c36c-4899-a060-907aa14dffad" containerID="594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794" exitCode=0
Sep 15 11:52:05.144082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.143907    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"
Sep 15 11:52:05.144082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.143932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb" event={"ID":"ed713dcd-c36c-4899-a060-907aa14dffad","Type":"ContainerDied","Data":"594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794"}
Sep 15 11:52:05.144082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.144067    2574 scope.go:117] "RemoveContainer" containerID="594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794"
Sep 15 11:52:05.145043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.144184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb" event={"ID":"ed713dcd-c36c-4899-a060-907aa14dffad","Type":"ContainerDied","Data":"f0bbb7098e33b68566c36e2abbe9a9ac1892614e97d7e8ccf738f8825fc92a9d"}
Sep 15 11:52:05.161853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.161835    2574 scope.go:117] "RemoveContainer" containerID="594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794"
Sep 15 11:52:05.162141 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:52:05.162121    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794\": container with ID starting with 594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794 not found: ID does not exist" containerID="594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794"
Sep 15 11:52:05.162191 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.162150    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794"} err="failed to get container status \"594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794\": rpc error: code = NotFound desc = could not find container \"594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794\": container with ID starting with 594e84ceea9ec046e2166fafff74e64f4e018f64bbb9ecaf0881879a773a6794 not found: ID does not exist"
Sep 15 11:52:05.173660 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.173615    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"]
Sep 15 11:52:05.177112 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:05.177089    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4dbbbf-67975c844b-vfjtb"]
Sep 15 11:52:06.666061 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:06.666029    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed713dcd-c36c-4899-a060-907aa14dffad" path="/var/lib/kubelet/pods/ed713dcd-c36c-4899-a060-907aa14dffad/volumes"
Sep 15 11:52:06.876819 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:06.876791    2574 scope.go:117] "RemoveContainer" containerID="14f79880ef31c00ac37ec792921ca0601edb25e6991fd109a9dc6620daf07a7b"
Sep 15 11:52:06.885955 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:06.885938    2574 scope.go:117] "RemoveContainer" containerID="651bb1e66030d8283add380f33283b802cc83b4fae7738b0e493512e2969507b"
Sep 15 11:52:06.893725 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:06.893707    2574 scope.go:117] "RemoveContainer" containerID="d75a29d11e549b3e06786d8467aef6149b3b3d523354604e6fa9eff4b92d566a"
Sep 15 11:52:13.127917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:13.127880    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"]
Sep 15 11:52:13.132226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:13.132196    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2fab1f-rzkzc"]
Sep 15 11:52:14.665593 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:14.665559    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d62a54-e294-440c-8c42-80c2421902c5" path="/var/lib/kubelet/pods/07d62a54-e294-440c-8c42-80c2421902c5/volumes"
Sep 15 11:52:17.127014 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.126943    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx" podUID="666e4036-8df4-40b8-8937-506770e4af1f" containerName="main" containerID="cri-o://8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465" gracePeriod=45
Sep 15 11:52:17.147970 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.147919    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9" podUID="ce1b92fe-d45f-4883-86b1-974250bc389c" containerName="main" containerID="cri-o://5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295" gracePeriod=45
Sep 15 11:52:17.157216 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.157181    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6" podUID="27d8e103-544d-4ba8-b52d-c57abe5c48b4" containerName="main" containerID="cri-o://adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d" gracePeriod=45
Sep 15 11:52:17.412194 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.412170    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:52:17.486483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.486454    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:52:17.490353 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.490329    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:52:17.544602 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544212    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-dshm\") pod \"ce1b92fe-d45f-4883-86b1-974250bc389c\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544829    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-home\") pod \"ce1b92fe-d45f-4883-86b1-974250bc389c\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544882    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgzxq\" (UniqueName: \"kubernetes.io/projected/666e4036-8df4-40b8-8937-506770e4af1f-kube-api-access-qgzxq\") pod \"666e4036-8df4-40b8-8937-506770e4af1f\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544915    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-dshm\") pod \"666e4036-8df4-40b8-8937-506770e4af1f\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544943    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-home\") pod \"666e4036-8df4-40b8-8937-506770e4af1f\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544969    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-model-cache\") pod \"ce1b92fe-d45f-4883-86b1-974250bc389c\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.544994    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-dshm\") pod \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") "
Sep 15 11:52:17.545106 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545039    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-tmp-dir\") pod \"ce1b92fe-d45f-4883-86b1-974250bc389c\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545606    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqdww\" (UniqueName: \"kubernetes.io/projected/27d8e103-544d-4ba8-b52d-c57abe5c48b4-kube-api-access-mqdww\") pod \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545686    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666e4036-8df4-40b8-8937-506770e4af1f-tls-certs\") pod \"666e4036-8df4-40b8-8937-506770e4af1f\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545731    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-model-cache\") pod \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545771    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tls-certs\") pod \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545807    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-home\") pod \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545843    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-model-cache\") pod \"666e4036-8df4-40b8-8937-506770e4af1f\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545873    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1b92fe-d45f-4883-86b1-974250bc389c-tls-certs\") pod \"ce1b92fe-d45f-4883-86b1-974250bc389c\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545900    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-tmp-dir\") pod \"666e4036-8df4-40b8-8937-506770e4af1f\" (UID: \"666e4036-8df4-40b8-8937-506770e4af1f\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545935    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tmp-dir\") pod \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\" (UID: \"27d8e103-544d-4ba8-b52d-c57abe5c48b4\") "
Sep 15 11:52:17.546445 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.545958    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-229zz\" (UniqueName: \"kubernetes.io/projected/ce1b92fe-d45f-4883-86b1-974250bc389c-kube-api-access-229zz\") pod \"ce1b92fe-d45f-4883-86b1-974250bc389c\" (UID: \"ce1b92fe-d45f-4883-86b1-974250bc389c\") "
Sep 15 11:52:17.547041 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.546542    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-home" (OuterVolumeSpecName: "home") pod "ce1b92fe-d45f-4883-86b1-974250bc389c" (UID: "ce1b92fe-d45f-4883-86b1-974250bc389c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.547041 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.546788    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-home" (OuterVolumeSpecName: "home") pod "666e4036-8df4-40b8-8937-506770e4af1f" (UID: "666e4036-8df4-40b8-8937-506770e4af1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.547382 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.547351    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce1b92fe-d45f-4883-86b1-974250bc389c" (UID: "ce1b92fe-d45f-4883-86b1-974250bc389c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.547490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.547440    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "666e4036-8df4-40b8-8937-506770e4af1f" (UID: "666e4036-8df4-40b8-8937-506770e4af1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.547610 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.547574    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "666e4036-8df4-40b8-8937-506770e4af1f" (UID: "666e4036-8df4-40b8-8937-506770e4af1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.547667 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.547634    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce1b92fe-d45f-4883-86b1-974250bc389c" (UID: "ce1b92fe-d45f-4883-86b1-974250bc389c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.547832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.547813    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "27d8e103-544d-4ba8-b52d-c57abe5c48b4" (UID: "27d8e103-544d-4ba8-b52d-c57abe5c48b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.550005 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.549714    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d8e103-544d-4ba8-b52d-c57abe5c48b4-kube-api-access-mqdww" (OuterVolumeSpecName: "kube-api-access-mqdww") pod "27d8e103-544d-4ba8-b52d-c57abe5c48b4" (UID: "27d8e103-544d-4ba8-b52d-c57abe5c48b4"). InnerVolumeSpecName "kube-api-access-mqdww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:17.551633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.551518    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666e4036-8df4-40b8-8937-506770e4af1f-kube-api-access-qgzxq" (OuterVolumeSpecName: "kube-api-access-qgzxq") pod "666e4036-8df4-40b8-8937-506770e4af1f" (UID: "666e4036-8df4-40b8-8937-506770e4af1f"). InnerVolumeSpecName "kube-api-access-qgzxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:17.551738 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.551669    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-home" (OuterVolumeSpecName: "home") pod "27d8e103-544d-4ba8-b52d-c57abe5c48b4" (UID: "27d8e103-544d-4ba8-b52d-c57abe5c48b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.552005 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.551962    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27d8e103-544d-4ba8-b52d-c57abe5c48b4" (UID: "27d8e103-544d-4ba8-b52d-c57abe5c48b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.552291 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.552266    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1b92fe-d45f-4883-86b1-974250bc389c-kube-api-access-229zz" (OuterVolumeSpecName: "kube-api-access-229zz") pod "ce1b92fe-d45f-4883-86b1-974250bc389c" (UID: "ce1b92fe-d45f-4883-86b1-974250bc389c"). InnerVolumeSpecName "kube-api-access-229zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:52:17.553066 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.553010    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-dshm" (OuterVolumeSpecName: "dshm") pod "ce1b92fe-d45f-4883-86b1-974250bc389c" (UID: "ce1b92fe-d45f-4883-86b1-974250bc389c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.553326 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.553279    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce1b92fe-d45f-4883-86b1-974250bc389c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce1b92fe-d45f-4883-86b1-974250bc389c" (UID: "ce1b92fe-d45f-4883-86b1-974250bc389c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:17.553408 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.553390    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27d8e103-544d-4ba8-b52d-c57abe5c48b4" (UID: "27d8e103-544d-4ba8-b52d-c57abe5c48b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:17.553579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.553557    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-dshm" (OuterVolumeSpecName: "dshm") pod "27d8e103-544d-4ba8-b52d-c57abe5c48b4" (UID: "27d8e103-544d-4ba8-b52d-c57abe5c48b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.553579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.553563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-dshm" (OuterVolumeSpecName: "dshm") pod "666e4036-8df4-40b8-8937-506770e4af1f" (UID: "666e4036-8df4-40b8-8937-506770e4af1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:52:17.553731 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.553712    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/666e4036-8df4-40b8-8937-506770e4af1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "666e4036-8df4-40b8-8937-506770e4af1f" (UID: "666e4036-8df4-40b8-8937-506770e4af1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:52:17.647262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647230    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647262 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647260    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647270    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgzxq\" (UniqueName: \"kubernetes.io/projected/666e4036-8df4-40b8-8937-506770e4af1f-kube-api-access-qgzxq\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647280    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647288    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647296    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647304    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647332    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce1b92fe-d45f-4883-86b1-974250bc389c-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647340    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqdww\" (UniqueName: \"kubernetes.io/projected/27d8e103-544d-4ba8-b52d-c57abe5c48b4-kube-api-access-mqdww\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647348    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/666e4036-8df4-40b8-8937-506770e4af1f-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647358    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647366    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647373    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647380    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647388    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce1b92fe-d45f-4883-86b1-974250bc389c-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647395    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/666e4036-8df4-40b8-8937-506770e4af1f-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647402    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27d8e103-544d-4ba8-b52d-c57abe5c48b4-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:17.647459 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:17.647409    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-229zz\" (UniqueName: \"kubernetes.io/projected/ce1b92fe-d45f-4883-86b1-974250bc389c-kube-api-access-229zz\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:52:18.222378 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.222342    2574 generic.go:356] "Generic (PLEG): container finished" podID="666e4036-8df4-40b8-8937-506770e4af1f" containerID="8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465" exitCode=0
Sep 15 11:52:18.222853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.222381    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"
Sep 15 11:52:18.222853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.222426    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx" event={"ID":"666e4036-8df4-40b8-8937-506770e4af1f","Type":"ContainerDied","Data":"8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465"}
Sep 15 11:52:18.222853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.222471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx" event={"ID":"666e4036-8df4-40b8-8937-506770e4af1f","Type":"ContainerDied","Data":"11d75527e1f4fe397957c6dec137835aac0348c47d112dacf47e467d1540392a"}
Sep 15 11:52:18.222853 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.222492    2574 scope.go:117] "RemoveContainer" containerID="8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465"
Sep 15 11:52:18.224060 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.223978    2574 generic.go:356] "Generic (PLEG): container finished" podID="ce1b92fe-d45f-4883-86b1-974250bc389c" containerID="5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295" exitCode=0
Sep 15 11:52:18.224060 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.224031    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"
Sep 15 11:52:18.224232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.224084    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9" event={"ID":"ce1b92fe-d45f-4883-86b1-974250bc389c","Type":"ContainerDied","Data":"5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295"}
Sep 15 11:52:18.224232 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.224126    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9" event={"ID":"ce1b92fe-d45f-4883-86b1-974250bc389c","Type":"ContainerDied","Data":"3af7515d894042f6c8e6edbd604af661d531244fe1e15f96198fa22239a944e2"}
Sep 15 11:52:18.225568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.225546    2574 generic.go:356] "Generic (PLEG): container finished" podID="27d8e103-544d-4ba8-b52d-c57abe5c48b4" containerID="adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d" exitCode=0
Sep 15 11:52:18.225670 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.225583    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"
Sep 15 11:52:18.225670 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.225622    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6" event={"ID":"27d8e103-544d-4ba8-b52d-c57abe5c48b4","Type":"ContainerDied","Data":"adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d"}
Sep 15 11:52:18.225670 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.225662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6" event={"ID":"27d8e103-544d-4ba8-b52d-c57abe5c48b4","Type":"ContainerDied","Data":"76b98af503176d324359dfe43397ba19ebce59f1038e5d6083182d2e7c852d78"}
Sep 15 11:52:18.234300 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.234284    2574 scope.go:117] "RemoveContainer" containerID="8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465"
Sep 15 11:52:18.234653 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:52:18.234634    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465\": container with ID starting with 8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465 not found: ID does not exist" containerID="8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465"
Sep 15 11:52:18.234752 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.234667    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465"} err="failed to get container status \"8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465\": rpc error: code = NotFound desc = could not find container \"8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465\": container with ID starting with 8c471daef355bf196a477594f220fa75b8d4b84c561b1d053505624bed597465 not found: ID does not exist"
Sep 15 11:52:18.234752 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.234688    2574 scope.go:117] "RemoveContainer" containerID="5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295"
Sep 15 11:52:18.243350 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.243328    2574 scope.go:117] "RemoveContainer" containerID="5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295"
Sep 15 11:52:18.243642 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:52:18.243622    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295\": container with ID starting with 5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295 not found: ID does not exist" containerID="5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295"
Sep 15 11:52:18.243717 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.243649    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295"} err="failed to get container status \"5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295\": rpc error: code = NotFound desc = could not find container \"5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295\": container with ID starting with 5bf521046a42fd9dfa89f99e59c15083ae101e94602a137277b4d3b4522b8295 not found: ID does not exist"
Sep 15 11:52:18.243717 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.243665    2574 scope.go:117] "RemoveContainer" containerID="adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d"
Sep 15 11:52:18.249726 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.249701    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"]
Sep 15 11:52:18.252204 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.252176    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/distinct-2-w4-2fab1f-kserve-6767d48c48-qbdxx"]
Sep 15 11:52:18.252962 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.252945    2574 scope.go:117] "RemoveContainer" containerID="adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d"
Sep 15 11:52:18.253250 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:52:18.253208    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d\": container with ID starting with adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d not found: ID does not exist" containerID="adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d"
Sep 15 11:52:18.253326 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.253261    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d"} err="failed to get container status \"adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d\": rpc error: code = NotFound desc = could not find container \"adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d\": container with ID starting with adac5b99d39870c57be14ddacfede6786e6ac00fcd5cf28e7bde62fbc2acdc7d not found: ID does not exist"
Sep 15 11:52:18.265217 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.265191    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"]
Sep 15 11:52:18.272531 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.272505    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/simulator-w4-2fab1f-kserve-78bb66cccc-t2gw6"]
Sep 15 11:52:18.283153 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.283093    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"]
Sep 15 11:52:18.285590 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.285556    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2fab1f/premium-w4-2fab1f-kserve-68d65bc648-dffr9"]
Sep 15 11:52:18.664730 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.664696    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d8e103-544d-4ba8-b52d-c57abe5c48b4" path="/var/lib/kubelet/pods/27d8e103-544d-4ba8-b52d-c57abe5c48b4/volumes"
Sep 15 11:52:18.665105 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.665093    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="666e4036-8df4-40b8-8937-506770e4af1f" path="/var/lib/kubelet/pods/666e4036-8df4-40b8-8937-506770e4af1f/volumes"
Sep 15 11:52:18.665457 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:52:18.665445    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1b92fe-d45f-4883-86b1-974250bc389c" path="/var/lib/kubelet/pods/ce1b92fe-d45f-4883-86b1-974250bc389c/volumes"
Sep 15 11:53:02.341747 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:02.341707    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"]
Sep 15 11:53:03.155554 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:03.155519    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"]
Sep 15 11:53:17.360904 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.360857    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84" podUID="55cc7119-2a52-4e2a-beed-174880730b72" containerName="main" containerID="cri-o://54ced14f2b02b146330a0e3a5587271482bcabeb97382648ba22be6972f5734f" gracePeriod=45
Sep 15 11:53:17.564158 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.564125    2574 generic.go:356] "Generic (PLEG): container finished" podID="55cc7119-2a52-4e2a-beed-174880730b72" containerID="54ced14f2b02b146330a0e3a5587271482bcabeb97382648ba22be6972f5734f" exitCode=0
Sep 15 11:53:17.564309 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.564195    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84" event={"ID":"55cc7119-2a52-4e2a-beed-174880730b72","Type":"ContainerDied","Data":"54ced14f2b02b146330a0e3a5587271482bcabeb97382648ba22be6972f5734f"}
Sep 15 11:53:17.617939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.617871    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:53:17.630206 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630179    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsq6\" (UniqueName: \"kubernetes.io/projected/55cc7119-2a52-4e2a-beed-174880730b72-kube-api-access-6wsq6\") pod \"55cc7119-2a52-4e2a-beed-174880730b72\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") "
Sep 15 11:53:17.630359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630214    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-home\") pod \"55cc7119-2a52-4e2a-beed-174880730b72\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") "
Sep 15 11:53:17.630359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630246    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-tmp-dir\") pod \"55cc7119-2a52-4e2a-beed-174880730b72\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") "
Sep 15 11:53:17.630359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630271    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-model-cache\") pod \"55cc7119-2a52-4e2a-beed-174880730b72\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") "
Sep 15 11:53:17.630359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630295    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-dshm\") pod \"55cc7119-2a52-4e2a-beed-174880730b72\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") "
Sep 15 11:53:17.630359 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630330    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc7119-2a52-4e2a-beed-174880730b72-tls-certs\") pod \"55cc7119-2a52-4e2a-beed-174880730b72\" (UID: \"55cc7119-2a52-4e2a-beed-174880730b72\") "
Sep 15 11:53:17.630658 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630484    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-home" (OuterVolumeSpecName: "home") pod "55cc7119-2a52-4e2a-beed-174880730b72" (UID: "55cc7119-2a52-4e2a-beed-174880730b72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:17.630658 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630537    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55cc7119-2a52-4e2a-beed-174880730b72" (UID: "55cc7119-2a52-4e2a-beed-174880730b72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:17.630658 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630602    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-model-cache" (OuterVolumeSpecName: "model-cache") pod "55cc7119-2a52-4e2a-beed-174880730b72" (UID: "55cc7119-2a52-4e2a-beed-174880730b72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:17.630834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630665    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:17.630834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.630682    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:17.632562 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.632538    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cc7119-2a52-4e2a-beed-174880730b72-kube-api-access-6wsq6" (OuterVolumeSpecName: "kube-api-access-6wsq6") pod "55cc7119-2a52-4e2a-beed-174880730b72" (UID: "55cc7119-2a52-4e2a-beed-174880730b72"). InnerVolumeSpecName "kube-api-access-6wsq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:53:17.632562 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.632549    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-dshm" (OuterVolumeSpecName: "dshm") pod "55cc7119-2a52-4e2a-beed-174880730b72" (UID: "55cc7119-2a52-4e2a-beed-174880730b72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:17.632700 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.632638    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cc7119-2a52-4e2a-beed-174880730b72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55cc7119-2a52-4e2a-beed-174880730b72" (UID: "55cc7119-2a52-4e2a-beed-174880730b72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:53:17.731797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.731762    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsq6\" (UniqueName: \"kubernetes.io/projected/55cc7119-2a52-4e2a-beed-174880730b72-kube-api-access-6wsq6\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:17.731797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.731792    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:17.731797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.731802    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc7119-2a52-4e2a-beed-174880730b72-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:17.731797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:17.731810    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc7119-2a52-4e2a-beed-174880730b72-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.175131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.175090    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q" podUID="daa7c146-09e5-424f-ad45-f241779f893b" containerName="main" containerID="cri-o://16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4" gracePeriod=45
Sep 15 11:53:18.431289 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.431229    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:53:18.440443 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440417    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7c146-09e5-424f-ad45-f241779f893b-tls-certs\") pod \"daa7c146-09e5-424f-ad45-f241779f893b\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") "
Sep 15 11:53:18.440633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440492    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-home\") pod \"daa7c146-09e5-424f-ad45-f241779f893b\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") "
Sep 15 11:53:18.440633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440551    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-model-cache\") pod \"daa7c146-09e5-424f-ad45-f241779f893b\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") "
Sep 15 11:53:18.440633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440576    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-tmp-dir\") pod \"daa7c146-09e5-424f-ad45-f241779f893b\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") "
Sep 15 11:53:18.440633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440606    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-dshm\") pod \"daa7c146-09e5-424f-ad45-f241779f893b\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") "
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440684    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjjkr\" (UniqueName: \"kubernetes.io/projected/daa7c146-09e5-424f-ad45-f241779f893b-kube-api-access-tjjkr\") pod \"daa7c146-09e5-424f-ad45-f241779f893b\" (UID: \"daa7c146-09e5-424f-ad45-f241779f893b\") "
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440739    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-home" (OuterVolumeSpecName: "home") pod "daa7c146-09e5-424f-ad45-f241779f893b" (UID: "daa7c146-09e5-424f-ad45-f241779f893b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440849    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "daa7c146-09e5-424f-ad45-f241779f893b" (UID: "daa7c146-09e5-424f-ad45-f241779f893b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.440881    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-model-cache" (OuterVolumeSpecName: "model-cache") pod "daa7c146-09e5-424f-ad45-f241779f893b" (UID: "daa7c146-09e5-424f-ad45-f241779f893b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.441068    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.441089    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.441157 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.441105    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.442984 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.442948    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa7c146-09e5-424f-ad45-f241779f893b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "daa7c146-09e5-424f-ad45-f241779f893b" (UID: "daa7c146-09e5-424f-ad45-f241779f893b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:53:18.443120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.443011    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa7c146-09e5-424f-ad45-f241779f893b-kube-api-access-tjjkr" (OuterVolumeSpecName: "kube-api-access-tjjkr") pod "daa7c146-09e5-424f-ad45-f241779f893b" (UID: "daa7c146-09e5-424f-ad45-f241779f893b"). InnerVolumeSpecName "kube-api-access-tjjkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:53:18.443120 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.443083    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-dshm" (OuterVolumeSpecName: "dshm") pod "daa7c146-09e5-424f-ad45-f241779f893b" (UID: "daa7c146-09e5-424f-ad45-f241779f893b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:53:18.542405 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.542364    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/daa7c146-09e5-424f-ad45-f241779f893b-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.542405 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.542397    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/daa7c146-09e5-424f-ad45-f241779f893b-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.542405 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.542406    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjkr\" (UniqueName: \"kubernetes.io/projected/daa7c146-09e5-424f-ad45-f241779f893b-kube-api-access-tjjkr\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:53:18.571813 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.571783    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"
Sep 15 11:53:18.571991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.571842    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84" event={"ID":"55cc7119-2a52-4e2a-beed-174880730b72","Type":"ContainerDied","Data":"99d03ac659410ba0cfd0d51ad35f4a7b1b7057a334bbf32c79a9c1c787b10d50"}
Sep 15 11:53:18.571991 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.571906    2574 scope.go:117] "RemoveContainer" containerID="54ced14f2b02b146330a0e3a5587271482bcabeb97382648ba22be6972f5734f"
Sep 15 11:53:18.573413 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.573387    2574 generic.go:356] "Generic (PLEG): container finished" podID="daa7c146-09e5-424f-ad45-f241779f893b" containerID="16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4" exitCode=0
Sep 15 11:53:18.573569 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.573430    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q" event={"ID":"daa7c146-09e5-424f-ad45-f241779f893b","Type":"ContainerDied","Data":"16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4"}
Sep 15 11:53:18.573569 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.573435    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"
Sep 15 11:53:18.573569 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.573451    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q" event={"ID":"daa7c146-09e5-424f-ad45-f241779f893b","Type":"ContainerDied","Data":"198f3f936e45f5c221715598fbeec8657388983cb102215d0fedbdb12e6bf244"}
Sep 15 11:53:18.583266 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.583247    2574 scope.go:117] "RemoveContainer" containerID="16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4"
Sep 15 11:53:18.591860 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.591840    2574 scope.go:117] "RemoveContainer" containerID="16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4"
Sep 15 11:53:18.592153 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:53:18.592130    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4\": container with ID starting with 16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4 not found: ID does not exist" containerID="16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4"
Sep 15 11:53:18.592234 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.592160    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4"} err="failed to get container status \"16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4\": rpc error: code = NotFound desc = could not find container \"16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4\": container with ID starting with 16fd6f41e09db2921da0a3c2bcdd501625c087c825f7784f3f5a505538c7f7b4 not found: ID does not exist"
Sep 15 11:53:18.598353 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.598328    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"]
Sep 15 11:53:18.602741 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.602673    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-26f1b0/auth-test-model-26f1b0-kserve-548dd7687-g4n84"]
Sep 15 11:53:18.614149 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.614127    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"]
Sep 15 11:53:18.619851 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.619831    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/auth-test-model-dd6959-kserve-7b78d764f5-hrd5q"]
Sep 15 11:53:18.665902 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.665866    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cc7119-2a52-4e2a-beed-174880730b72" path="/var/lib/kubelet/pods/55cc7119-2a52-4e2a-beed-174880730b72/volumes"
Sep 15 11:53:18.666278 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:18.666263    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa7c146-09e5-424f-ad45-f241779f893b" path="/var/lib/kubelet/pods/daa7c146-09e5-424f-ad45-f241779f893b/volumes"
Sep 15 11:53:56.072202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072162    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"]
Sep 15 11:53:56.072659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072606    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed713dcd-c36c-4899-a060-907aa14dffad" containerName="payload-pre-processing"
Sep 15 11:53:56.072659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072619    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed713dcd-c36c-4899-a060-907aa14dffad" containerName="payload-pre-processing"
Sep 15 11:53:56.072659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072632    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d8e103-544d-4ba8-b52d-c57abe5c48b4" containerName="main"
Sep 15 11:53:56.072659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072638    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d8e103-544d-4ba8-b52d-c57abe5c48b4" containerName="main"
Sep 15 11:53:56.072659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072648    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="666e4036-8df4-40b8-8937-506770e4af1f" containerName="main"
Sep 15 11:53:56.072659 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072653    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="666e4036-8df4-40b8-8937-506770e4af1f" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072670    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1b92fe-d45f-4883-86b1-974250bc389c" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072679    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1b92fe-d45f-4883-86b1-974250bc389c" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072693    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cc7119-2a52-4e2a-beed-174880730b72" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072698    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cc7119-2a52-4e2a-beed-174880730b72" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072710    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d62a54-e294-440c-8c42-80c2421902c5" containerName="revoke-keys"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072715    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d62a54-e294-440c-8c42-80c2421902c5" containerName="revoke-keys"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072725    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa7c146-09e5-424f-ad45-f241779f893b" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072730    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa7c146-09e5-424f-ad45-f241779f893b" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072782    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d62a54-e294-440c-8c42-80c2421902c5" containerName="revoke-keys"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072790    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cc7119-2a52-4e2a-beed-174880730b72" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072797    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1b92fe-d45f-4883-86b1-974250bc389c" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072804    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed713dcd-c36c-4899-a060-907aa14dffad" containerName="payload-pre-processing"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072811    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa7c146-09e5-424f-ad45-f241779f893b" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072817    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d8e103-544d-4ba8-b52d-c57abe5c48b4" containerName="main"
Sep 15 11:53:56.072908 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.072824    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="666e4036-8df4-40b8-8937-506770e4af1f" containerName="main"
Sep 15 11:53:56.076040 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.076009    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.078913 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.078894    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"kube-root-ca.crt\""
Sep 15 11:53:56.079402 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.079377    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"sub-test-model-dd6959-kserve-self-signed-certs\""
Sep 15 11:53:56.079545 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.079405    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"openshift-service-ca.crt\""
Sep 15 11:53:56.079545 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.079430    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"default-dockercfg-2gsgm\""
Sep 15 11:53:56.085702 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.085681    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"]
Sep 15 11:53:56.094664 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.094635    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5rln\" (UniqueName: \"kubernetes.io/projected/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-kube-api-access-p5rln\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.094773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.094671    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-home\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.094773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.094696    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tmp-dir\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.094773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.094718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tls-certs\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.094773 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.094769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-dshm\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.094905 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.094847    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-model-cache\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195418 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195386    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-model-cache\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5rln\" (UniqueName: \"kubernetes.io/projected/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-kube-api-access-p5rln\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-home\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tmp-dir\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195629 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195620    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tls-certs\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195885 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-dshm\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195885 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195838    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-home\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.195885 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195872    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-model-cache\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.196056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.195956    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tmp-dir\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.197926 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.197901    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-dshm\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.198196 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.198179    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tls-certs\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.203978 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.203957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5rln\" (UniqueName: \"kubernetes.io/projected/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-kube-api-access-p5rln\") pod \"sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.385971 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.385886    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:53:56.657943 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.657878    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"]
Sep 15 11:53:56.661092 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:53:56.661061    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb93bea_5ee1_4196_ac03_25aa3a585b5b.slice/crio-775a7c21cef15918fabb50abd1a67eeb3bc9254eefc00787e24dd2343302ac0c WatchSource:0}: Error finding container 775a7c21cef15918fabb50abd1a67eeb3bc9254eefc00787e24dd2343302ac0c: Status 404 returned error can't find the container with id 775a7c21cef15918fabb50abd1a67eeb3bc9254eefc00787e24dd2343302ac0c
Sep 15 11:53:56.773914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.773881    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv" event={"ID":"9bb93bea-5ee1-4196-ac03-25aa3a585b5b","Type":"ContainerStarted","Data":"3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519"}
Sep 15 11:53:56.773914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.773928    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv" event={"ID":"9bb93bea-5ee1-4196-ac03-25aa3a585b5b","Type":"ContainerStarted","Data":"775a7c21cef15918fabb50abd1a67eeb3bc9254eefc00787e24dd2343302ac0c"}
Sep 15 11:53:56.789579 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:53:56.789535    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv" podStartSLOduration=0.789516435 podStartE2EDuration="789.516435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:53:56.788835864 +0000 UTC m=+1730.566512120" watchObservedRunningTime="2026-09-15 11:53:56.789516435 +0000 UTC m=+1730.567192685"
Sep 15 11:54:06.386057 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:06.386018    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:54:06.386667 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:06.386077    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:54:06.399150 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:06.399120    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:54:06.840366 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:06.840337    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:54:06.957823 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:06.957792    2574 scope.go:117] "RemoveContainer" containerID="37f57e885715691f773e2e5450d7eaf862ef4ba289f793718a1701ea7df67c5d"
Sep 15 11:54:38.553298 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:38.553263    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"]
Sep 15 11:54:53.572789 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.572729    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv" podUID="9bb93bea-5ee1-4196-ac03-25aa3a585b5b" containerName="main" containerID="cri-o://3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519" gracePeriod=45
Sep 15 11:54:53.811914 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.811891    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:54:53.945384 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945353    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-dshm\") pod \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") "
Sep 15 11:54:53.945574 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945412    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-home\") pod \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") "
Sep 15 11:54:53.945574 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945429    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-model-cache\") pod \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") "
Sep 15 11:54:53.945574 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945483    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tls-certs\") pod \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") "
Sep 15 11:54:53.945574 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945556    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5rln\" (UniqueName: \"kubernetes.io/projected/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-kube-api-access-p5rln\") pod \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") "
Sep 15 11:54:53.945741 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tmp-dir\") pod \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\" (UID: \"9bb93bea-5ee1-4196-ac03-25aa3a585b5b\") "
Sep 15 11:54:53.945741 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945683    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-home" (OuterVolumeSpecName: "home") pod "9bb93bea-5ee1-4196-ac03-25aa3a585b5b" (UID: "9bb93bea-5ee1-4196-ac03-25aa3a585b5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:53.945741 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945710    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bb93bea-5ee1-4196-ac03-25aa3a585b5b" (UID: "9bb93bea-5ee1-4196-ac03-25aa3a585b5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:53.945934 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945909    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bb93bea-5ee1-4196-ac03-25aa3a585b5b" (UID: "9bb93bea-5ee1-4196-ac03-25aa3a585b5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:53.946038 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945962    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:54:53.946038 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.945975    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:54:53.947570 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.947543    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-dshm" (OuterVolumeSpecName: "dshm") pod "9bb93bea-5ee1-4196-ac03-25aa3a585b5b" (UID: "9bb93bea-5ee1-4196-ac03-25aa3a585b5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:54:53.947656 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.947571    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bb93bea-5ee1-4196-ac03-25aa3a585b5b" (UID: "9bb93bea-5ee1-4196-ac03-25aa3a585b5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:54:53.947697 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:53.947665    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-kube-api-access-p5rln" (OuterVolumeSpecName: "kube-api-access-p5rln") pod "9bb93bea-5ee1-4196-ac03-25aa3a585b5b" (UID: "9bb93bea-5ee1-4196-ac03-25aa3a585b5b"). InnerVolumeSpecName "kube-api-access-p5rln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:54:54.047153 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.047115    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:54:54.047153 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.047143    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:54:54.047153 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.047154    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:54:54.047377 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.047166    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5rln\" (UniqueName: \"kubernetes.io/projected/9bb93bea-5ee1-4196-ac03-25aa3a585b5b-kube-api-access-p5rln\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:54:54.075365 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.075325    2574 generic.go:356] "Generic (PLEG): container finished" podID="9bb93bea-5ee1-4196-ac03-25aa3a585b5b" containerID="3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519" exitCode=0
Sep 15 11:54:54.075551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.075370    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"
Sep 15 11:54:54.075551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.075413    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv" event={"ID":"9bb93bea-5ee1-4196-ac03-25aa3a585b5b","Type":"ContainerDied","Data":"3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519"}
Sep 15 11:54:54.075551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.075464    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv" event={"ID":"9bb93bea-5ee1-4196-ac03-25aa3a585b5b","Type":"ContainerDied","Data":"775a7c21cef15918fabb50abd1a67eeb3bc9254eefc00787e24dd2343302ac0c"}
Sep 15 11:54:54.075551 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.075486    2574 scope.go:117] "RemoveContainer" containerID="3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519"
Sep 15 11:54:54.085394 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.085379    2574 scope.go:117] "RemoveContainer" containerID="3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519"
Sep 15 11:54:54.085730 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:54:54.085712    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519\": container with ID starting with 3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519 not found: ID does not exist" containerID="3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519"
Sep 15 11:54:54.085790 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.085739    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519"} err="failed to get container status \"3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519\": rpc error: code = NotFound desc = could not find container \"3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519\": container with ID starting with 3e30b9b21f8fcf30cc0d37791aa8ed949cbefb013e24fb8c95926cc83fede519 not found: ID does not exist"
Sep 15 11:54:54.097172 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.097148    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"]
Sep 15 11:54:54.100446 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.100424    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/sub-test-model-dd6959-kserve-6c86bb5c97-h7rhv"]
Sep 15 11:54:54.665633 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:54:54.665598    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb93bea-5ee1-4196-ac03-25aa3a585b5b" path="/var/lib/kubelet/pods/9bb93bea-5ee1-4196-ac03-25aa3a585b5b/volumes"
Sep 15 11:55:06.767196 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:06.767170    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:55:06.771556 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:06.771530    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 11:55:06.771845 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:06.771828    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:55:06.775944 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:06.775924    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 11:55:21.378771 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.378735    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"]
Sep 15 11:55:21.379388 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.379371    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb93bea-5ee1-4196-ac03-25aa3a585b5b" containerName="main"
Sep 15 11:55:21.379434 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.379391    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb93bea-5ee1-4196-ac03-25aa3a585b5b" containerName="main"
Sep 15 11:55:21.379544 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.379533    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bb93bea-5ee1-4196-ac03-25aa3a585b5b" containerName="main"
Sep 15 11:55:21.382915 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.382896    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.386039 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.386012    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"openshift-service-ca.crt\""
Sep 15 11:55:21.386177 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.386013    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"rate-test-model-dd6959-kserve-self-signed-certs\""
Sep 15 11:55:21.386177 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.386131    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"default-dockercfg-2gsgm\""
Sep 15 11:55:21.386297 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.386024    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-dd6959\"/\"kube-root-ca.crt\""
Sep 15 11:55:21.391876 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.391850    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"]
Sep 15 11:55:21.408177 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.408148    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-model-cache\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.408322 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.408196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4j5t\" (UniqueName: \"kubernetes.io/projected/9c0a405c-8ad0-4567-a504-62f3b3a976ab-kube-api-access-s4j5t\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.408322 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.408251    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-dshm\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.408322 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.408278    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tmp-dir\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.408322 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.408314    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-home\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.408492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.408342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tls-certs\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.509566 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.509530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tls-certs\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.509719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.509647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-model-cache\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.509719 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.509707    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4j5t\" (UniqueName: \"kubernetes.io/projected/9c0a405c-8ad0-4567-a504-62f3b3a976ab-kube-api-access-s4j5t\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.509834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.509732    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-dshm\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.509834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.509753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tmp-dir\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.509834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.509792    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-home\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.510081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.510060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-model-cache\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.510159 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.510138    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tmp-dir\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.510201 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.510176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-home\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.511949 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.511923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-dshm\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.512179 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.512160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tls-certs\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.517788 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.517766    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4j5t\" (UniqueName: \"kubernetes.io/projected/9c0a405c-8ad0-4567-a504-62f3b3a976ab-kube-api-access-s4j5t\") pod \"rate-test-model-dd6959-kserve-97695864c-mztwg\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") " pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.695190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.695154    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:21.968696 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:21.968671    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"]
Sep 15 11:55:21.971091 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:55:21.971065    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c0a405c_8ad0_4567_a504_62f3b3a976ab.slice/crio-eb4dfbc213b25d5b9c076a6e311f2084ee12f79eb87ef1be43793983c3ff2e5c WatchSource:0}: Error finding container eb4dfbc213b25d5b9c076a6e311f2084ee12f79eb87ef1be43793983c3ff2e5c: Status 404 returned error can't find the container with id eb4dfbc213b25d5b9c076a6e311f2084ee12f79eb87ef1be43793983c3ff2e5c
Sep 15 11:55:22.226964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:22.226879    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg" event={"ID":"9c0a405c-8ad0-4567-a504-62f3b3a976ab","Type":"ContainerStarted","Data":"e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9"}
Sep 15 11:55:22.226964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:22.226924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg" event={"ID":"9c0a405c-8ad0-4567-a504-62f3b3a976ab","Type":"ContainerStarted","Data":"eb4dfbc213b25d5b9c076a6e311f2084ee12f79eb87ef1be43793983c3ff2e5c"}
Sep 15 11:55:22.244760 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:22.244709    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg" podStartSLOduration=1.244694299 podStartE2EDuration="1.244694299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:55:22.242742065 +0000 UTC m=+1816.020418322" watchObservedRunningTime="2026-09-15 11:55:22.244694299 +0000 UTC m=+1816.022370606"
Sep 15 11:55:31.695707 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:31.695661    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:31.696175 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:31.695745    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:31.708983 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:31.708955    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:32.293240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:32.293212    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:55:56.761416 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:55:56.761387    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"]
Sep 15 11:56:03.964280 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:03.964246    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"]
Sep 15 11:56:03.968088 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:03.968067    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:03.974964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:03.974930    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0ecfa6-dockercfg-4j9m2\""
Sep 15 11:56:03.975461 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:03.975440    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0ecfa6\""
Sep 15 11:56:03.986710 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:03.986686    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"]
Sep 15 11:56:04.016280 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.016250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05da19f-f8cc-4881-bf0a-da537fdec1a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.016280 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.016283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggspp\" (UniqueName: \"kubernetes.io/projected/f05da19f-f8cc-4881-bf0a-da537fdec1a9-kube-api-access-ggspp\") pod \"payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.117413 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.117286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05da19f-f8cc-4881-bf0a-da537fdec1a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.117413 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.117327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggspp\" (UniqueName: \"kubernetes.io/projected/f05da19f-f8cc-4881-bf0a-da537fdec1a9-kube-api-access-ggspp\") pod \"payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.122619 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.118075    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05da19f-f8cc-4881-bf0a-da537fdec1a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.129832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.129806    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggspp\" (UniqueName: \"kubernetes.io/projected/f05da19f-f8cc-4881-bf0a-da537fdec1a9-kube-api-access-ggspp\") pod \"payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.281286 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.281194    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:04.575636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:04.575612    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"]
Sep 15 11:56:04.578092 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:56:04.578063    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05da19f_f8cc_4881_bf0a_da537fdec1a9.slice/crio-d9e117276378b3a52161ac7429471852ed4f9d7985ee8bd606d4032f9a031358 WatchSource:0}: Error finding container d9e117276378b3a52161ac7429471852ed4f9d7985ee8bd606d4032f9a031358: Status 404 returned error can't find the container with id d9e117276378b3a52161ac7429471852ed4f9d7985ee8bd606d4032f9a031358
Sep 15 11:56:05.458465 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:05.458427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f" event={"ID":"f05da19f-f8cc-4881-bf0a-da537fdec1a9","Type":"ContainerStarted","Data":"77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e"}
Sep 15 11:56:05.458874 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:05.458489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f" event={"ID":"f05da19f-f8cc-4881-bf0a-da537fdec1a9","Type":"ContainerStarted","Data":"d9e117276378b3a52161ac7429471852ed4f9d7985ee8bd606d4032f9a031358"}
Sep 15 11:56:05.458874 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:05.458583    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:05.490142 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:05.490085    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f" podStartSLOduration=2.490069344 podStartE2EDuration="2.490069344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:05.489240202 +0000 UTC m=+1859.266916471" watchObservedRunningTime="2026-09-15 11:56:05.490069344 +0000 UTC m=+1859.267745602"
Sep 15 11:56:06.997268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:06.997236    2574 scope.go:117] "RemoveContainer" containerID="660619d1e5532e6407bf3ebaf9de94977d74b81c05926eb36e7f85afed6e2497"
Sep 15 11:56:11.782270 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:11.782202    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg" podUID="9c0a405c-8ad0-4567-a504-62f3b3a976ab" containerName="main" containerID="cri-o://e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9" gracePeriod=45
Sep 15 11:56:12.029110 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.029081    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:56:12.092190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092102    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tmp-dir\") pod \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") "
Sep 15 11:56:12.092190 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092167    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-dshm\") pod \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") "
Sep 15 11:56:12.092421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092196    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-model-cache\") pod \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") "
Sep 15 11:56:12.092421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092226    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tls-certs\") pod \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") "
Sep 15 11:56:12.092421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s4j5t\" (UniqueName: \"kubernetes.io/projected/9c0a405c-8ad0-4567-a504-62f3b3a976ab-kube-api-access-s4j5t\") pod \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") "
Sep 15 11:56:12.092421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092314    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-home\") pod \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\" (UID: \"9c0a405c-8ad0-4567-a504-62f3b3a976ab\") "
Sep 15 11:56:12.092693 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092434    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c0a405c-8ad0-4567-a504-62f3b3a976ab" (UID: "9c0a405c-8ad0-4567-a504-62f3b3a976ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:12.092693 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092455    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c0a405c-8ad0-4567-a504-62f3b3a976ab" (UID: "9c0a405c-8ad0-4567-a504-62f3b3a976ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:12.092693 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092602    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-home" (OuterVolumeSpecName: "home") pod "9c0a405c-8ad0-4567-a504-62f3b3a976ab" (UID: "9c0a405c-8ad0-4567-a504-62f3b3a976ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:12.092893 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092873    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:56:12.092947 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092899    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:56:12.092947 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.092915    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:56:12.094407 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.094376    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0a405c-8ad0-4567-a504-62f3b3a976ab-kube-api-access-s4j5t" (OuterVolumeSpecName: "kube-api-access-s4j5t") pod "9c0a405c-8ad0-4567-a504-62f3b3a976ab" (UID: "9c0a405c-8ad0-4567-a504-62f3b3a976ab"). InnerVolumeSpecName "kube-api-access-s4j5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:56:12.094509 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.094433    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-dshm" (OuterVolumeSpecName: "dshm") pod "9c0a405c-8ad0-4567-a504-62f3b3a976ab" (UID: "9c0a405c-8ad0-4567-a504-62f3b3a976ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:56:12.094806 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.094781    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c0a405c-8ad0-4567-a504-62f3b3a976ab" (UID: "9c0a405c-8ad0-4567-a504-62f3b3a976ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:56:12.194256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.194225    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c0a405c-8ad0-4567-a504-62f3b3a976ab-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:56:12.194256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.194253    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c0a405c-8ad0-4567-a504-62f3b3a976ab-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:56:12.194256 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.194265    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4j5t\" (UniqueName: \"kubernetes.io/projected/9c0a405c-8ad0-4567-a504-62f3b3a976ab-kube-api-access-s4j5t\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:56:12.498930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.498889    2574 generic.go:356] "Generic (PLEG): container finished" podID="9c0a405c-8ad0-4567-a504-62f3b3a976ab" containerID="e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9" exitCode=0
Sep 15 11:56:12.499131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.498937    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"
Sep 15 11:56:12.499131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.498974    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg" event={"ID":"9c0a405c-8ad0-4567-a504-62f3b3a976ab","Type":"ContainerDied","Data":"e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9"}
Sep 15 11:56:12.499131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.499022    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg" event={"ID":"9c0a405c-8ad0-4567-a504-62f3b3a976ab","Type":"ContainerDied","Data":"eb4dfbc213b25d5b9c076a6e311f2084ee12f79eb87ef1be43793983c3ff2e5c"}
Sep 15 11:56:12.499131 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.499041    2574 scope.go:117] "RemoveContainer" containerID="e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9"
Sep 15 11:56:12.508834 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.508812    2574 scope.go:117] "RemoveContainer" containerID="e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9"
Sep 15 11:56:12.509132 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:56:12.509114    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9\": container with ID starting with e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9 not found: ID does not exist" containerID="e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9"
Sep 15 11:56:12.509175 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.509142    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9"} err="failed to get container status \"e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9\": rpc error: code = NotFound desc = could not find container \"e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9\": container with ID starting with e5cb61094c1de0909ed108875a1701525810d9652568fc118a8800f1dc3e9ff9 not found: ID does not exist"
Sep 15 11:56:12.519749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.519716    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"]
Sep 15 11:56:12.524200 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.524172    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-dd6959/rate-test-model-dd6959-kserve-97695864c-mztwg"]
Sep 15 11:56:12.665707 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:12.665675    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0a405c-8ad0-4567-a504-62f3b3a976ab" path="/var/lib/kubelet/pods/9c0a405c-8ad0-4567-a504-62f3b3a976ab/volumes"
Sep 15 11:56:16.466250 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:16.466223    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:56:34.343787 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.343711    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"]
Sep 15 11:56:34.344274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.344139    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0a405c-8ad0-4567-a504-62f3b3a976ab" containerName="main"
Sep 15 11:56:34.344274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.344152    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0a405c-8ad0-4567-a504-62f3b3a976ab" containerName="main"
Sep 15 11:56:34.344274 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.344210    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0a405c-8ad0-4567-a504-62f3b3a976ab" containerName="main"
Sep 15 11:56:34.348542 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.348526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.351424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.351401    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2023d5\""
Sep 15 11:56:34.351424 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.351401    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2023d5-dockercfg-jbzrd\""
Sep 15 11:56:34.355439 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.355416    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"]
Sep 15 11:56:34.374216 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.374186    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"]
Sep 15 11:56:34.378133 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.378109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.390081 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.390053    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"]
Sep 15 11:56:34.509454 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.509410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fdbcad-af38-43b9-bc5e-44bf080c472a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.509721 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.509466    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf56x\" (UniqueName: \"kubernetes.io/projected/e8fdbcad-af38-43b9-bc5e-44bf080c472a-kube-api-access-rf56x\") pod \"payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.509721 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.509637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74c6\" (UniqueName: \"kubernetes.io/projected/dbd46c9b-8c60-44d3-a846-788ff0d0607a-kube-api-access-n74c6\") pod \"payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.509721 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.509688    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd46c9b-8c60-44d3-a846-788ff0d0607a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.610420 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.610323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74c6\" (UniqueName: \"kubernetes.io/projected/dbd46c9b-8c60-44d3-a846-788ff0d0607a-kube-api-access-n74c6\") pod \"payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.610420 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.610375    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd46c9b-8c60-44d3-a846-788ff0d0607a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.610694 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.610547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fdbcad-af38-43b9-bc5e-44bf080c472a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.610694 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.610578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf56x\" (UniqueName: \"kubernetes.io/projected/e8fdbcad-af38-43b9-bc5e-44bf080c472a-kube-api-access-rf56x\") pod \"payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.611285 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.611262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd46c9b-8c60-44d3-a846-788ff0d0607a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.611415 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.611340    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fdbcad-af38-43b9-bc5e-44bf080c472a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.618342 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.618317    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf56x\" (UniqueName: \"kubernetes.io/projected/e8fdbcad-af38-43b9-bc5e-44bf080c472a-kube-api-access-rf56x\") pod \"payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.618470 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.618404    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74c6\" (UniqueName: \"kubernetes.io/projected/dbd46c9b-8c60-44d3-a846-788ff0d0607a-kube-api-access-n74c6\") pod \"payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.659421 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.659387    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:34.689171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.689144    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:34.936998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.936974    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"]
Sep 15 11:56:34.938552 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:56:34.938525    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd46c9b_8c60_44d3_a846_788ff0d0607a.slice/crio-14672ad313870762d3df3dea9d5fd11e716ebfc2980662b3bead01da3ac3e266 WatchSource:0}: Error finding container 14672ad313870762d3df3dea9d5fd11e716ebfc2980662b3bead01da3ac3e266: Status 404 returned error can't find the container with id 14672ad313870762d3df3dea9d5fd11e716ebfc2980662b3bead01da3ac3e266
Sep 15 11:56:34.980419 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:34.980396    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"]
Sep 15 11:56:34.992757 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:56:34.992730    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fdbcad_af38_43b9_bc5e_44bf080c472a.slice/crio-9f01cb73d73bcac656ce7718ad116081e2ed0c33df4894f67c8ae502c2eb50b3 WatchSource:0}: Error finding container 9f01cb73d73bcac656ce7718ad116081e2ed0c33df4894f67c8ae502c2eb50b3: Status 404 returned error can't find the container with id 9f01cb73d73bcac656ce7718ad116081e2ed0c33df4894f67c8ae502c2eb50b3
Sep 15 11:56:35.622370 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.622332    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn" event={"ID":"e8fdbcad-af38-43b9-bc5e-44bf080c472a","Type":"ContainerStarted","Data":"f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326"}
Sep 15 11:56:35.622937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.622395    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn" event={"ID":"e8fdbcad-af38-43b9-bc5e-44bf080c472a","Type":"ContainerStarted","Data":"9f01cb73d73bcac656ce7718ad116081e2ed0c33df4894f67c8ae502c2eb50b3"}
Sep 15 11:56:35.622937 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.622669    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:35.624080 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.624055    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q" event={"ID":"dbd46c9b-8c60-44d3-a846-788ff0d0607a","Type":"ContainerStarted","Data":"114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7"}
Sep 15 11:56:35.624223 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.624094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q" event={"ID":"dbd46c9b-8c60-44d3-a846-788ff0d0607a","Type":"ContainerStarted","Data":"14672ad313870762d3df3dea9d5fd11e716ebfc2980662b3bead01da3ac3e266"}
Sep 15 11:56:35.624223 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.624143    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:56:35.658392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.658330    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn" podStartSLOduration=1.6583139980000001 podStartE2EDuration="1.658313998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:35.655880281 +0000 UTC m=+1889.433556540" watchObservedRunningTime="2026-09-15 11:56:35.658313998 +0000 UTC m=+1889.435990255"
Sep 15 11:56:35.690946 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:35.690889    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q" podStartSLOduration=1.690868906 podStartE2EDuration="1.690868906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:56:35.688622238 +0000 UTC m=+1889.466298496" watchObservedRunningTime="2026-09-15 11:56:35.690868906 +0000 UTC m=+1889.468545164"
Sep 15 11:56:46.632154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:46.632122    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:56:46.632597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:56:46.632194    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:57:01.859519 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:01.859465    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn_e8fdbcad-af38-43b9-bc5e-44bf080c472a/payload-processing/0.log"
Sep 15 11:57:06.370935 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.370887    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"]
Sep 15 11:57:06.374850 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.374828    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.377272 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.377250    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"openshift-service-ca.crt\""
Sep 15 11:57:06.377396 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.377280    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"ipp-route-0ecfa6-kserve-self-signed-certs\""
Sep 15 11:57:06.377980 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.377962    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"kube-root-ca.crt\""
Sep 15 11:57:06.378087 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.378047    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"default-dockercfg-l9r64\""
Sep 15 11:57:06.384354 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.384326    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"]
Sep 15 11:57:06.529091 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.529043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-tmp-dir\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.529303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.529103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-home\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.529303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.529143    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wssvq\" (UniqueName: \"kubernetes.io/projected/b93e006a-680e-438a-87a2-82e10a843413-kube-api-access-wssvq\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.529303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.529211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e006a-680e-438a-87a2-82e10a843413-tls-certs\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.529303 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.529285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-model-cache\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.529482 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.529307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-dshm\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630468 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-model-cache\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630695 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630529    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-dshm\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630695 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-tmp-dir\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630825 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-home\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630825 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wssvq\" (UniqueName: \"kubernetes.io/projected/b93e006a-680e-438a-87a2-82e10a843413-kube-api-access-wssvq\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e006a-680e-438a-87a2-82e10a843413-tls-certs\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.630930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-model-cache\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.631047 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.630985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-home\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.631099 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.631040    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-tmp-dir\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.632797 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.632776    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-dshm\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.633123 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.633109    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"ipp-route-0ecfa6-kserve-self-signed-certs\""
Sep 15 11:57:06.638134 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.638116    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"kube-root-ca.crt\""
Sep 15 11:57:06.643195 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.643177    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e006a-680e-438a-87a2-82e10a843413-tls-certs\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.648050 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.648034    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"openshift-service-ca.crt\""
Sep 15 11:57:06.658925 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.658900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wssvq\" (UniqueName: \"kubernetes.io/projected/b93e006a-680e-438a-87a2-82e10a843413-kube-api-access-wssvq\") pod \"ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") " pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.688878 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.688848    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-0ecfa6\"/\"default-dockercfg-l9r64\""
Sep 15 11:57:06.697573 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.697551    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:06.971524 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:06.971481    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"]
Sep 15 11:57:06.973126 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:57:06.973098    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb93e006a_680e_438a_87a2_82e10a843413.slice/crio-2f28944cf825104f1dc9035f03c944e7f77037bb8e769e2ff31f211a132d9d65 WatchSource:0}: Error finding container 2f28944cf825104f1dc9035f03c944e7f77037bb8e769e2ff31f211a132d9d65: Status 404 returned error can't find the container with id 2f28944cf825104f1dc9035f03c944e7f77037bb8e769e2ff31f211a132d9d65
Sep 15 11:57:07.020863 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:07.020839    2574 scope.go:117] "RemoveContainer" containerID="7ff78e795b30d9055a6ac2503abebc7f5fe058bf3e77de35376b7e9a2da261a8"
Sep 15 11:57:07.797976 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:07.797931    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp" event={"ID":"b93e006a-680e-438a-87a2-82e10a843413","Type":"ContainerStarted","Data":"6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15"}
Sep 15 11:57:07.798437 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:07.797987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp" event={"ID":"b93e006a-680e-438a-87a2-82e10a843413","Type":"ContainerStarted","Data":"2f28944cf825104f1dc9035f03c944e7f77037bb8e769e2ff31f211a132d9d65"}
Sep 15 11:57:07.815078 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:07.815022    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp" podStartSLOduration=1.815005374 podStartE2EDuration="1.815005374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:07.813796899 +0000 UTC m=+1921.591473355" watchObservedRunningTime="2026-09-15 11:57:07.815005374 +0000 UTC m=+1921.592681630"
Sep 15 11:57:16.698456 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:16.698418    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:16.698887 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:16.698470    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:16.711513 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:16.711476    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:16.856130 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:16.856098    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:57:32.819678 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:32.819638    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn_e8fdbcad-af38-43b9-bc5e-44bf080c472a/payload-processing/0.log"
Sep 15 11:57:40.416286 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.416246    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"]
Sep 15 11:57:40.420871 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.420853    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.423374 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.423350    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-cd36d2\""
Sep 15 11:57:40.423492 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.423355    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-cd36d2-dockercfg-2mkkx\""
Sep 15 11:57:40.430832 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.430799    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"]
Sep 15 11:57:40.553965 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.553924    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcwq\" (UniqueName: \"kubernetes.io/projected/47212302-debf-4bb8-b008-2555e31cec0b-kube-api-access-mlcwq\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.554129 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.553986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47212302-debf-4bb8-b008-2555e31cec0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.655168 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.655128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcwq\" (UniqueName: \"kubernetes.io/projected/47212302-debf-4bb8-b008-2555e31cec0b-kube-api-access-mlcwq\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.655362 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.655189    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47212302-debf-4bb8-b008-2555e31cec0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.655934 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.655913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47212302-debf-4bb8-b008-2555e31cec0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.663090 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.663066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcwq\" (UniqueName: \"kubernetes.io/projected/47212302-debf-4bb8-b008-2555e31cec0b-kube-api-access-mlcwq\") pod \"payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:40.732720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:40.732642    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:41.006082 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:41.006053    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"]
Sep 15 11:57:41.007626 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:57:41.007601    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47212302_debf_4bb8_b008_2555e31cec0b.slice/crio-c45ce22121faf14f9e600f76ff5d6f6e590106b09d2b827985591f5bf11078a5 WatchSource:0}: Error finding container c45ce22121faf14f9e600f76ff5d6f6e590106b09d2b827985591f5bf11078a5: Status 404 returned error can't find the container with id c45ce22121faf14f9e600f76ff5d6f6e590106b09d2b827985591f5bf11078a5
Sep 15 11:57:41.974059 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:41.974026    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh" event={"ID":"47212302-debf-4bb8-b008-2555e31cec0b","Type":"ContainerStarted","Data":"c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a"}
Sep 15 11:57:41.974467 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:41.974077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh" event={"ID":"47212302-debf-4bb8-b008-2555e31cec0b","Type":"ContainerStarted","Data":"c45ce22121faf14f9e600f76ff5d6f6e590106b09d2b827985591f5bf11078a5"}
Sep 15 11:57:41.974467 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:41.974118    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:57:41.991636 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:41.991585    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh" podStartSLOduration=1.991569286 podStartE2EDuration="1.991569286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:57:41.989827072 +0000 UTC m=+1955.767503330" watchObservedRunningTime="2026-09-15 11:57:41.991569286 +0000 UTC m=+1955.769245543"
Sep 15 11:57:52.981796 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:57:52.981766    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:58:06.896602 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:06.896523    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"]
Sep 15 11:58:06.900486 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:06.900469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:06.902479 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:06.902458    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:58:06.902621 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:06.902484    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:58:06.907634 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:06.907606    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"]
Sep 15 11:58:07.015875 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.015843    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74387ab2-bdfc-4d67-abd7-254e67adb31d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj\" (UID: \"74387ab2-bdfc-4d67-abd7-254e67adb31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:07.048743 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.048713    2574 scope.go:117] "RemoveContainer" containerID="cf0566917502b22a348e6d3615abe31006bc9501e42d1874535d6c6a321c4121"
Sep 15 11:58:07.057171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.057145    2574 scope.go:117] "RemoveContainer" containerID="98b24165df50f4ccc1f63e18304923a312f36cad9d6cb26133e9684b676ce219"
Sep 15 11:58:07.065008 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.064991    2574 scope.go:117] "RemoveContainer" containerID="5d800d418556881a78e2ee6ea548ed34b3501671a1754172e3e4dce3bc26d00a"
Sep 15 11:58:07.072820 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.072803    2574 scope.go:117] "RemoveContainer" containerID="5d83eed308364339f251403e27f0d9021fef0891ccdcbd02394e94db6a0c7ddc"
Sep 15 11:58:07.080033 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.080016    2574 scope.go:117] "RemoveContainer" containerID="b424a1c3f4cac85c21d5be9ca41008baa684d3380f1c17f507f77bcbdf616e38"
Sep 15 11:58:07.116981 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.116952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74387ab2-bdfc-4d67-abd7-254e67adb31d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj\" (UID: \"74387ab2-bdfc-4d67-abd7-254e67adb31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:07.117557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.117539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74387ab2-bdfc-4d67-abd7-254e67adb31d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj\" (UID: \"74387ab2-bdfc-4d67-abd7-254e67adb31d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:07.214988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.214898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:07.486557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:07.486530    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"]
Sep 15 11:58:07.488696 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:58:07.488663    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74387ab2_bdfc_4d67_abd7_254e67adb31d.slice/crio-e1aff7500592bed839ad37e1155b4271b6b7b9ffb3005d3103e66e05882082ce WatchSource:0}: Error finding container e1aff7500592bed839ad37e1155b4271b6b7b9ffb3005d3103e66e05882082ce: Status 404 returned error can't find the container with id e1aff7500592bed839ad37e1155b4271b6b7b9ffb3005d3103e66e05882082ce
Sep 15 11:58:08.120438 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:08.120359    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj" event={"ID":"74387ab2-bdfc-4d67-abd7-254e67adb31d","Type":"ContainerStarted","Data":"16e895691f1a15c5c68994c6e3e8098e98b4c3ca89adddfceda03a7cabff75e6"}
Sep 15 11:58:08.120438 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:08.120411    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj" event={"ID":"74387ab2-bdfc-4d67-abd7-254e67adb31d","Type":"ContainerStarted","Data":"e1aff7500592bed839ad37e1155b4271b6b7b9ffb3005d3103e66e05882082ce"}
Sep 15 11:58:08.157221 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:08.157168    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj" podStartSLOduration=2.157150608 podStartE2EDuration="2.157150608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 11:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 11:58:08.155022143 +0000 UTC m=+1981.932698401" watchObservedRunningTime="2026-09-15 11:58:08.157150608 +0000 UTC m=+1981.934826887"
Sep 15 11:58:09.126988 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:09.126951    2574 generic.go:356] "Generic (PLEG): container finished" podID="74387ab2-bdfc-4d67-abd7-254e67adb31d" containerID="16e895691f1a15c5c68994c6e3e8098e98b4c3ca89adddfceda03a7cabff75e6" exitCode=0
Sep 15 11:58:09.127373 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:09.127033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj" event={"ID":"74387ab2-bdfc-4d67-abd7-254e67adb31d","Type":"ContainerDied","Data":"16e895691f1a15c5c68994c6e3e8098e98b4c3ca89adddfceda03a7cabff75e6"}
Sep 15 11:58:10.266709 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:10.266688    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:10.349040 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:10.349008    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74387ab2-bdfc-4d67-abd7-254e67adb31d-maas-api-service-ca\") pod \"74387ab2-bdfc-4d67-abd7-254e67adb31d\" (UID: \"74387ab2-bdfc-4d67-abd7-254e67adb31d\") "
Sep 15 11:58:10.349370 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:10.349347    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74387ab2-bdfc-4d67-abd7-254e67adb31d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74387ab2-bdfc-4d67-abd7-254e67adb31d" (UID: "74387ab2-bdfc-4d67-abd7-254e67adb31d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:10.450269 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:10.450233    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74387ab2-bdfc-4d67-abd7-254e67adb31d-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:58:11.140483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:11.140452    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"
Sep 15 11:58:11.140708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:11.140455    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj" event={"ID":"74387ab2-bdfc-4d67-abd7-254e67adb31d","Type":"ContainerDied","Data":"e1aff7500592bed839ad37e1155b4271b6b7b9ffb3005d3103e66e05882082ce"}
Sep 15 11:58:11.140708 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:11.140583    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1aff7500592bed839ad37e1155b4271b6b7b9ffb3005d3103e66e05882082ce"
Sep 15 11:58:18.114990 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.114956    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"]
Sep 15 11:58:18.115560 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.115263    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh" podUID="47212302-debf-4bb8-b008-2555e31cec0b" containerName="payload-pre-processing" containerID="cri-o://c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a" gracePeriod=30
Sep 15 11:58:18.337652 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.337611    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"]
Sep 15 11:58:18.341180 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.341152    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-cd36d2-w6glj"]
Sep 15 11:58:18.375202 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.375154    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:58:18.531406 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.531367    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlcwq\" (UniqueName: \"kubernetes.io/projected/47212302-debf-4bb8-b008-2555e31cec0b-kube-api-access-mlcwq\") pod \"47212302-debf-4bb8-b008-2555e31cec0b\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") "
Sep 15 11:58:18.531603 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.531431    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47212302-debf-4bb8-b008-2555e31cec0b-plugins-config-volume\") pod \"47212302-debf-4bb8-b008-2555e31cec0b\" (UID: \"47212302-debf-4bb8-b008-2555e31cec0b\") "
Sep 15 11:58:18.531817 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.531787    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47212302-debf-4bb8-b008-2555e31cec0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "47212302-debf-4bb8-b008-2555e31cec0b" (UID: "47212302-debf-4bb8-b008-2555e31cec0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:18.533547 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.533489    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47212302-debf-4bb8-b008-2555e31cec0b-kube-api-access-mlcwq" (OuterVolumeSpecName: "kube-api-access-mlcwq") pod "47212302-debf-4bb8-b008-2555e31cec0b" (UID: "47212302-debf-4bb8-b008-2555e31cec0b"). InnerVolumeSpecName "kube-api-access-mlcwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:58:18.632985 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.632896    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlcwq\" (UniqueName: \"kubernetes.io/projected/47212302-debf-4bb8-b008-2555e31cec0b-kube-api-access-mlcwq\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:58:18.632985 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.632931    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47212302-debf-4bb8-b008-2555e31cec0b-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:58:18.665887 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:18.665856    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74387ab2-bdfc-4d67-abd7-254e67adb31d" path="/var/lib/kubelet/pods/74387ab2-bdfc-4d67-abd7-254e67adb31d/volumes"
Sep 15 11:58:19.180568 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.180531    2574 generic.go:356] "Generic (PLEG): container finished" podID="47212302-debf-4bb8-b008-2555e31cec0b" containerID="c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a" exitCode=0
Sep 15 11:58:19.181048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.180634    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"
Sep 15 11:58:19.181048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.180634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh" event={"ID":"47212302-debf-4bb8-b008-2555e31cec0b","Type":"ContainerDied","Data":"c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a"}
Sep 15 11:58:19.181048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.180762    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh" event={"ID":"47212302-debf-4bb8-b008-2555e31cec0b","Type":"ContainerDied","Data":"c45ce22121faf14f9e600f76ff5d6f6e590106b09d2b827985591f5bf11078a5"}
Sep 15 11:58:19.181048 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.180790    2574 scope.go:117] "RemoveContainer" containerID="c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a"
Sep 15 11:58:19.189917 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.189899    2574 scope.go:117] "RemoveContainer" containerID="c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a"
Sep 15 11:58:19.190189 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:58:19.190172    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a\": container with ID starting with c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a not found: ID does not exist" containerID="c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a"
Sep 15 11:58:19.190240 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.190198    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a"} err="failed to get container status \"c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a\": rpc error: code = NotFound desc = could not find container \"c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a\": container with ID starting with c86b9f53c2de7fcb2099966ce0346acc482e0d096aa63f02b01ec91be71c7f5a not found: ID does not exist"
Sep 15 11:58:19.196015 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.195992    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"]
Sep 15 11:58:19.197966 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:19.197947    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-cd36d2-57cd56cf8-785gh"]
Sep 15 11:58:20.665226 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:20.665189    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47212302-debf-4bb8-b008-2555e31cec0b" path="/var/lib/kubelet/pods/47212302-debf-4bb8-b008-2555e31cec0b/volumes"
Sep 15 11:58:28.634810 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.634777    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"]
Sep 15 11:58:28.635388 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.635368    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74387ab2-bdfc-4d67-abd7-254e67adb31d" containerName="revoke-keys"
Sep 15 11:58:28.635490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.635390    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="74387ab2-bdfc-4d67-abd7-254e67adb31d" containerName="revoke-keys"
Sep 15 11:58:28.635490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.635421    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47212302-debf-4bb8-b008-2555e31cec0b" containerName="payload-pre-processing"
Sep 15 11:58:28.635490 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.635431    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47212302-debf-4bb8-b008-2555e31cec0b" containerName="payload-pre-processing"
Sep 15 11:58:28.635673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.635560    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47212302-debf-4bb8-b008-2555e31cec0b" containerName="payload-pre-processing"
Sep 15 11:58:28.635673 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.635575    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="74387ab2-bdfc-4d67-abd7-254e67adb31d" containerName="revoke-keys"
Sep 15 11:58:28.640560 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.640541    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:28.643321 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.643273    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:58:28.643321 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.643296    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:58:28.645763 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.645734    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"]
Sep 15 11:58:28.730089 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.730059    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2936205-e24a-4510-90f0-d74c1232ceb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75\" (UID: \"b2936205-e24a-4510-90f0-d74c1232ceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:28.831413 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.831381    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2936205-e24a-4510-90f0-d74c1232ceb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75\" (UID: \"b2936205-e24a-4510-90f0-d74c1232ceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:28.832111 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.832082    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2936205-e24a-4510-90f0-d74c1232ceb6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75\" (UID: \"b2936205-e24a-4510-90f0-d74c1232ceb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:28.952605 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:28.952560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:29.221948 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:29.221918    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"]
Sep 15 11:58:29.224258 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:58:29.224217    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2936205_e24a_4510_90f0_d74c1232ceb6.slice/crio-e13ab7d1a5643d62d27ef868f79244e7b44af7eb3f56112e9a4a142775b1c7af WatchSource:0}: Error finding container e13ab7d1a5643d62d27ef868f79244e7b44af7eb3f56112e9a4a142775b1c7af: Status 404 returned error can't find the container with id e13ab7d1a5643d62d27ef868f79244e7b44af7eb3f56112e9a4a142775b1c7af
Sep 15 11:58:29.236241 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:29.236214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75" event={"ID":"b2936205-e24a-4510-90f0-d74c1232ceb6","Type":"ContainerStarted","Data":"e13ab7d1a5643d62d27ef868f79244e7b44af7eb3f56112e9a4a142775b1c7af"}
Sep 15 11:58:30.242843 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:30.242759    2574 generic.go:356] "Generic (PLEG): container finished" podID="b2936205-e24a-4510-90f0-d74c1232ceb6" containerID="071211fe051c6d3119013b99994620dfa94e3d267773d158da00d1d8ddb5bfd5" exitCode=0
Sep 15 11:58:30.243187 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:30.242843    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75" event={"ID":"b2936205-e24a-4510-90f0-d74c1232ceb6","Type":"ContainerDied","Data":"071211fe051c6d3119013b99994620dfa94e3d267773d158da00d1d8ddb5bfd5"}
Sep 15 11:58:31.383584 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:31.383562    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:31.457154 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:31.457124    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2936205-e24a-4510-90f0-d74c1232ceb6-maas-api-service-ca\") pod \"b2936205-e24a-4510-90f0-d74c1232ceb6\" (UID: \"b2936205-e24a-4510-90f0-d74c1232ceb6\") "
Sep 15 11:58:31.457483 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:31.457460    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2936205-e24a-4510-90f0-d74c1232ceb6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2936205-e24a-4510-90f0-d74c1232ceb6" (UID: "b2936205-e24a-4510-90f0-d74c1232ceb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:31.558278 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:31.558192    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2936205-e24a-4510-90f0-d74c1232ceb6-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:58:32.253930 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:32.253903    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"
Sep 15 11:58:32.254095 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:32.253903    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75" event={"ID":"b2936205-e24a-4510-90f0-d74c1232ceb6","Type":"ContainerDied","Data":"e13ab7d1a5643d62d27ef868f79244e7b44af7eb3f56112e9a4a142775b1c7af"}
Sep 15 11:58:32.254095 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:32.254020    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e13ab7d1a5643d62d27ef868f79244e7b44af7eb3f56112e9a4a142775b1c7af"
Sep 15 11:58:44.248557 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.248520    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"]
Sep 15 11:58:44.248975 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.248773    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f" podUID="f05da19f-f8cc-4881-bf0a-da537fdec1a9" containerName="payload-pre-processing" containerID="cri-o://77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e" gracePeriod=30
Sep 15 11:58:44.438435 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.438404    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"]
Sep 15 11:58:44.442846 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.442818    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0ecfa6-9kt75"]
Sep 15 11:58:44.506665 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.506601    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:58:44.583866 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.583833    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05da19f-f8cc-4881-bf0a-da537fdec1a9-plugins-config-volume\") pod \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") "
Sep 15 11:58:44.584042 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.583904    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggspp\" (UniqueName: \"kubernetes.io/projected/f05da19f-f8cc-4881-bf0a-da537fdec1a9-kube-api-access-ggspp\") pod \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\" (UID: \"f05da19f-f8cc-4881-bf0a-da537fdec1a9\") "
Sep 15 11:58:44.584299 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.584264    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05da19f-f8cc-4881-bf0a-da537fdec1a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f05da19f-f8cc-4881-bf0a-da537fdec1a9" (UID: "f05da19f-f8cc-4881-bf0a-da537fdec1a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:58:44.586005 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.585979    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05da19f-f8cc-4881-bf0a-da537fdec1a9-kube-api-access-ggspp" (OuterVolumeSpecName: "kube-api-access-ggspp") pod "f05da19f-f8cc-4881-bf0a-da537fdec1a9" (UID: "f05da19f-f8cc-4881-bf0a-da537fdec1a9"). InnerVolumeSpecName "kube-api-access-ggspp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:58:44.666444 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.666412    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2936205-e24a-4510-90f0-d74c1232ceb6" path="/var/lib/kubelet/pods/b2936205-e24a-4510-90f0-d74c1232ceb6/volumes"
Sep 15 11:58:44.685009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.684984    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f05da19f-f8cc-4881-bf0a-da537fdec1a9-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:58:44.685009 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:44.685008    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggspp\" (UniqueName: \"kubernetes.io/projected/f05da19f-f8cc-4881-bf0a-da537fdec1a9-kube-api-access-ggspp\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:58:45.324363 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.324328    2574 generic.go:356] "Generic (PLEG): container finished" podID="f05da19f-f8cc-4881-bf0a-da537fdec1a9" containerID="77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e" exitCode=0
Sep 15 11:58:45.324801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.324377    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"
Sep 15 11:58:45.324801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.324396    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f" event={"ID":"f05da19f-f8cc-4881-bf0a-da537fdec1a9","Type":"ContainerDied","Data":"77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e"}
Sep 15 11:58:45.324801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.324437    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f" event={"ID":"f05da19f-f8cc-4881-bf0a-da537fdec1a9","Type":"ContainerDied","Data":"d9e117276378b3a52161ac7429471852ed4f9d7985ee8bd606d4032f9a031358"}
Sep 15 11:58:45.324801 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.324458    2574 scope.go:117] "RemoveContainer" containerID="77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e"
Sep 15 11:58:45.333898 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.333879    2574 scope.go:117] "RemoveContainer" containerID="77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e"
Sep 15 11:58:45.334177 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:58:45.334158    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e\": container with ID starting with 77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e not found: ID does not exist" containerID="77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e"
Sep 15 11:58:45.334220 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.334186    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e"} err="failed to get container status \"77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e\": rpc error: code = NotFound desc = could not find container \"77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e\": container with ID starting with 77093e297e5d047d309ea147520688aea021fa43831a9088906ddf2263eb4c5e not found: ID does not exist"
Sep 15 11:58:45.340205 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.340176    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"]
Sep 15 11:58:45.343850 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:45.343825    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0ecfa6-f58759bf4-n648f"]
Sep 15 11:58:46.666055 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:46.666023    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f05da19f-f8cc-4881-bf0a-da537fdec1a9" path="/var/lib/kubelet/pods/f05da19f-f8cc-4881-bf0a-da537fdec1a9/volumes"
Sep 15 11:58:49.757776 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:58:49.757736    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"]
Sep 15 11:59:04.777309 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:04.777250    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp" podUID="b93e006a-680e-438a-87a2-82e10a843413" containerName="main" containerID="cri-o://6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15" gracePeriod=45
Sep 15 11:59:05.034713 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.034651    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:59:05.087655 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087624    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e006a-680e-438a-87a2-82e10a843413-tls-certs\") pod \"b93e006a-680e-438a-87a2-82e10a843413\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") "
Sep 15 11:59:05.087818 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087707    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-home\") pod \"b93e006a-680e-438a-87a2-82e10a843413\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") "
Sep 15 11:59:05.087818 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-tmp-dir\") pod \"b93e006a-680e-438a-87a2-82e10a843413\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") "
Sep 15 11:59:05.087936 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087828    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wssvq\" (UniqueName: \"kubernetes.io/projected/b93e006a-680e-438a-87a2-82e10a843413-kube-api-access-wssvq\") pod \"b93e006a-680e-438a-87a2-82e10a843413\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") "
Sep 15 11:59:05.087936 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087856    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-dshm\") pod \"b93e006a-680e-438a-87a2-82e10a843413\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") "
Sep 15 11:59:05.087936 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087890    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-model-cache\") pod \"b93e006a-680e-438a-87a2-82e10a843413\" (UID: \"b93e006a-680e-438a-87a2-82e10a843413\") "
Sep 15 11:59:05.088107 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.087993    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-home" (OuterVolumeSpecName: "home") pod "b93e006a-680e-438a-87a2-82e10a843413" (UID: "b93e006a-680e-438a-87a2-82e10a843413"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:59:05.088107 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.088095    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b93e006a-680e-438a-87a2-82e10a843413" (UID: "b93e006a-680e-438a-87a2-82e10a843413"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:59:05.088231 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.088206    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-model-cache" (OuterVolumeSpecName: "model-cache") pod "b93e006a-680e-438a-87a2-82e10a843413" (UID: "b93e006a-680e-438a-87a2-82e10a843413"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:59:05.088324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.088306    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:05.088379 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.088327    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:05.088379 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.088340    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:05.090268 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.090237    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b93e006a-680e-438a-87a2-82e10a843413-kube-api-access-wssvq" (OuterVolumeSpecName: "kube-api-access-wssvq") pod "b93e006a-680e-438a-87a2-82e10a843413" (UID: "b93e006a-680e-438a-87a2-82e10a843413"). InnerVolumeSpecName "kube-api-access-wssvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:59:05.090447 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.090422    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b93e006a-680e-438a-87a2-82e10a843413-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b93e006a-680e-438a-87a2-82e10a843413" (UID: "b93e006a-680e-438a-87a2-82e10a843413"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 11:59:05.090578 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.090513    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-dshm" (OuterVolumeSpecName: "dshm") pod "b93e006a-680e-438a-87a2-82e10a843413" (UID: "b93e006a-680e-438a-87a2-82e10a843413"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 11:59:05.189640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.189599    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wssvq\" (UniqueName: \"kubernetes.io/projected/b93e006a-680e-438a-87a2-82e10a843413-kube-api-access-wssvq\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:05.189640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.189631    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b93e006a-680e-438a-87a2-82e10a843413-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:05.189640 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.189642    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b93e006a-680e-438a-87a2-82e10a843413-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:05.434385 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.434351    2574 generic.go:356] "Generic (PLEG): container finished" podID="b93e006a-680e-438a-87a2-82e10a843413" containerID="6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15" exitCode=0
Sep 15 11:59:05.434385 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.434391    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"
Sep 15 11:59:05.434703 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.434435    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp" event={"ID":"b93e006a-680e-438a-87a2-82e10a843413","Type":"ContainerDied","Data":"6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15"}
Sep 15 11:59:05.434703 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.434485    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp" event={"ID":"b93e006a-680e-438a-87a2-82e10a843413","Type":"ContainerDied","Data":"2f28944cf825104f1dc9035f03c944e7f77037bb8e769e2ff31f211a132d9d65"}
Sep 15 11:59:05.434703 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.434532    2574 scope.go:117] "RemoveContainer" containerID="6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15"
Sep 15 11:59:05.444022 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.444006    2574 scope.go:117] "RemoveContainer" containerID="6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15"
Sep 15 11:59:05.444261 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:59:05.444239    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15\": container with ID starting with 6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15 not found: ID does not exist" containerID="6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15"
Sep 15 11:59:05.444324 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.444276    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15"} err="failed to get container status \"6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15\": rpc error: code = NotFound desc = could not find container \"6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15\": container with ID starting with 6faa8784bf4d2a0f2bf3cb2dcbbfacd649f146af0a2d33e0dfdd6c51de2d1a15 not found: ID does not exist"
Sep 15 11:59:05.456046 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.456024    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"]
Sep 15 11:59:05.461488 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:05.461465    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-0ecfa6/ipp-route-0ecfa6-kserve-78cfd8fd99-7q6vp"]
Sep 15 11:59:06.665621 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:06.665589    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b93e006a-680e-438a-87a2-82e10a843413" path="/var/lib/kubelet/pods/b93e006a-680e-438a-87a2-82e10a843413/volumes"
Sep 15 11:59:33.872738 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.872654    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"]
Sep 15 11:59:33.873225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873095    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b93e006a-680e-438a-87a2-82e10a843413" containerName="main"
Sep 15 11:59:33.873225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873108    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b93e006a-680e-438a-87a2-82e10a843413" containerName="main"
Sep 15 11:59:33.873225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873132    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f05da19f-f8cc-4881-bf0a-da537fdec1a9" containerName="payload-pre-processing"
Sep 15 11:59:33.873225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873138    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f05da19f-f8cc-4881-bf0a-da537fdec1a9" containerName="payload-pre-processing"
Sep 15 11:59:33.873225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873148    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2936205-e24a-4510-90f0-d74c1232ceb6" containerName="revoke-keys"
Sep 15 11:59:33.873225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873157    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2936205-e24a-4510-90f0-d74c1232ceb6" containerName="revoke-keys"
Sep 15 11:59:33.873430 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873240    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2936205-e24a-4510-90f0-d74c1232ceb6" containerName="revoke-keys"
Sep 15 11:59:33.873430 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873251    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f05da19f-f8cc-4881-bf0a-da537fdec1a9" containerName="payload-pre-processing"
Sep 15 11:59:33.873430 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.873258    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b93e006a-680e-438a-87a2-82e10a843413" containerName="main"
Sep 15 11:59:33.876562 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.876542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:33.878993 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.878970    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:59:33.879155 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.879130    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:59:33.886472 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.886449    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"]
Sep 15 11:59:33.964927 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:33.964889    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428adb06-e9c7-4747-9561-b55f7931c6aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz\" (UID: \"428adb06-e9c7-4747-9561-b55f7931c6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:34.065466 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:34.065430    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428adb06-e9c7-4747-9561-b55f7931c6aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz\" (UID: \"428adb06-e9c7-4747-9561-b55f7931c6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:34.066056 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:34.066037    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428adb06-e9c7-4747-9561-b55f7931c6aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz\" (UID: \"428adb06-e9c7-4747-9561-b55f7931c6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:34.187248 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:34.187206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:34.461113 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:34.461088    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"]
Sep 15 11:59:34.463368 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:59:34.463340    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428adb06_e9c7_4747_9561_b55f7931c6aa.slice/crio-1443546459a6f83bd41f6fd94d88318d1eb80a73c6fb97d22eab25e73f7e78fa WatchSource:0}: Error finding container 1443546459a6f83bd41f6fd94d88318d1eb80a73c6fb97d22eab25e73f7e78fa: Status 404 returned error can't find the container with id 1443546459a6f83bd41f6fd94d88318d1eb80a73c6fb97d22eab25e73f7e78fa
Sep 15 11:59:34.587684 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:34.587647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz" event={"ID":"428adb06-e9c7-4747-9561-b55f7931c6aa","Type":"ContainerStarted","Data":"1443546459a6f83bd41f6fd94d88318d1eb80a73c6fb97d22eab25e73f7e78fa"}
Sep 15 11:59:35.595043 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:35.595004    2574 generic.go:356] "Generic (PLEG): container finished" podID="428adb06-e9c7-4747-9561-b55f7931c6aa" containerID="4ba7a52f5a858a924bca6385d93cbddc507621d12416215c7b8b64d10606c58f" exitCode=0
Sep 15 11:59:35.595395 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:35.595044    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz" event={"ID":"428adb06-e9c7-4747-9561-b55f7931c6aa","Type":"ContainerDied","Data":"4ba7a52f5a858a924bca6385d93cbddc507621d12416215c7b8b64d10606c58f"}
Sep 15 11:59:36.738229 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:36.738199    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:36.892680 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:36.892650    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428adb06-e9c7-4747-9561-b55f7931c6aa-maas-api-service-ca\") pod \"428adb06-e9c7-4747-9561-b55f7931c6aa\" (UID: \"428adb06-e9c7-4747-9561-b55f7931c6aa\") "
Sep 15 11:59:36.892996 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:36.892975    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428adb06-e9c7-4747-9561-b55f7931c6aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "428adb06-e9c7-4747-9561-b55f7931c6aa" (UID: "428adb06-e9c7-4747-9561-b55f7931c6aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:36.994309 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:36.994271    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428adb06-e9c7-4747-9561-b55f7931c6aa-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:37.606812 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:37.606780    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"
Sep 15 11:59:37.606998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:37.606818    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz" event={"ID":"428adb06-e9c7-4747-9561-b55f7931c6aa","Type":"ContainerDied","Data":"1443546459a6f83bd41f6fd94d88318d1eb80a73c6fb97d22eab25e73f7e78fa"}
Sep 15 11:59:37.606998 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:37.606866    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1443546459a6f83bd41f6fd94d88318d1eb80a73c6fb97d22eab25e73f7e78fa"
Sep 15 11:59:44.319308 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.319272    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"]
Sep 15 11:59:44.319783 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.319546    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn" podUID="e8fdbcad-af38-43b9-bc5e-44bf080c472a" containerName="payload-processing" containerID="cri-o://f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326" gracePeriod=30
Sep 15 11:59:44.325221 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.325188    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"]
Sep 15 11:59:44.325519 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.325447    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q" podUID="dbd46c9b-8c60-44d3-a846-788ff0d0607a" containerName="payload-pre-processing" containerID="cri-o://114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7" gracePeriod=30
Sep 15 11:59:44.471225 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.471149    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"]
Sep 15 11:59:44.473780 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.473747    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2023d5-lr4mz"]
Sep 15 11:59:44.603538 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.603486    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:59:44.606667 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.606645    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:59:44.645543 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.645491    2574 generic.go:356] "Generic (PLEG): container finished" podID="e8fdbcad-af38-43b9-bc5e-44bf080c472a" containerID="f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326" exitCode=0
Sep 15 11:59:44.645720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.645565    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"
Sep 15 11:59:44.645720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.645593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn" event={"ID":"e8fdbcad-af38-43b9-bc5e-44bf080c472a","Type":"ContainerDied","Data":"f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326"}
Sep 15 11:59:44.645720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.645660    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn" event={"ID":"e8fdbcad-af38-43b9-bc5e-44bf080c472a","Type":"ContainerDied","Data":"9f01cb73d73bcac656ce7718ad116081e2ed0c33df4894f67c8ae502c2eb50b3"}
Sep 15 11:59:44.645720 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.645696    2574 scope.go:117] "RemoveContainer" containerID="f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326"
Sep 15 11:59:44.646964 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.646940    2574 generic.go:356] "Generic (PLEG): container finished" podID="dbd46c9b-8c60-44d3-a846-788ff0d0607a" containerID="114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7" exitCode=0
Sep 15 11:59:44.647076 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.646977    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"
Sep 15 11:59:44.647076 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.646985    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q" event={"ID":"dbd46c9b-8c60-44d3-a846-788ff0d0607a","Type":"ContainerDied","Data":"114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7"}
Sep 15 11:59:44.647076 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.647025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q" event={"ID":"dbd46c9b-8c60-44d3-a846-788ff0d0607a","Type":"ContainerDied","Data":"14672ad313870762d3df3dea9d5fd11e716ebfc2980662b3bead01da3ac3e266"}
Sep 15 11:59:44.654901 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.654882    2574 scope.go:117] "RemoveContainer" containerID="f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326"
Sep 15 11:59:44.655194 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:59:44.655174    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326\": container with ID starting with f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326 not found: ID does not exist" containerID="f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326"
Sep 15 11:59:44.655253 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.655203    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326"} err="failed to get container status \"f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326\": rpc error: code = NotFound desc = could not find container \"f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326\": container with ID starting with f76113e1cfbbae7eb6e91336ac15746577bddf73fa289f68ecbe14d14d938326 not found: ID does not exist"
Sep 15 11:59:44.655253 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.655219    2574 scope.go:117] "RemoveContainer" containerID="114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7"
Sep 15 11:59:44.664222 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.664180    2574 scope.go:117] "RemoveContainer" containerID="114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7"
Sep 15 11:59:44.664478 ip-10-0-142-54 kubenswrapper[2574]: E0915 11:59:44.664460    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7\": container with ID starting with 114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7 not found: ID does not exist" containerID="114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7"
Sep 15 11:59:44.664617 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.664483    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7"} err="failed to get container status \"114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7\": rpc error: code = NotFound desc = could not find container \"114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7\": container with ID starting with 114b993eb013b3f147db7b4e30863edde0537abf99b3d93853ce20ff8365a0e7 not found: ID does not exist"
Sep 15 11:59:44.668451 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.668428    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428adb06-e9c7-4747-9561-b55f7931c6aa" path="/var/lib/kubelet/pods/428adb06-e9c7-4747-9561-b55f7931c6aa/volumes"
Sep 15 11:59:44.762986 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.762955    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd46c9b-8c60-44d3-a846-788ff0d0607a-plugins-config-volume\") pod \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") "
Sep 15 11:59:44.763160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.762994    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf56x\" (UniqueName: \"kubernetes.io/projected/e8fdbcad-af38-43b9-bc5e-44bf080c472a-kube-api-access-rf56x\") pod \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") "
Sep 15 11:59:44.763160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.763033    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fdbcad-af38-43b9-bc5e-44bf080c472a-plugins-config-volume\") pod \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\" (UID: \"e8fdbcad-af38-43b9-bc5e-44bf080c472a\") "
Sep 15 11:59:44.763160 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.763061    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n74c6\" (UniqueName: \"kubernetes.io/projected/dbd46c9b-8c60-44d3-a846-788ff0d0607a-kube-api-access-n74c6\") pod \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\" (UID: \"dbd46c9b-8c60-44d3-a846-788ff0d0607a\") "
Sep 15 11:59:44.763436 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.763412    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd46c9b-8c60-44d3-a846-788ff0d0607a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbd46c9b-8c60-44d3-a846-788ff0d0607a" (UID: "dbd46c9b-8c60-44d3-a846-788ff0d0607a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:44.763533 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.763449    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fdbcad-af38-43b9-bc5e-44bf080c472a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8fdbcad-af38-43b9-bc5e-44bf080c472a" (UID: "e8fdbcad-af38-43b9-bc5e-44bf080c472a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:44.765186 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.765157    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fdbcad-af38-43b9-bc5e-44bf080c472a-kube-api-access-rf56x" (OuterVolumeSpecName: "kube-api-access-rf56x") pod "e8fdbcad-af38-43b9-bc5e-44bf080c472a" (UID: "e8fdbcad-af38-43b9-bc5e-44bf080c472a"). InnerVolumeSpecName "kube-api-access-rf56x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:59:44.765295 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.765250    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd46c9b-8c60-44d3-a846-788ff0d0607a-kube-api-access-n74c6" (OuterVolumeSpecName: "kube-api-access-n74c6") pod "dbd46c9b-8c60-44d3-a846-788ff0d0607a" (UID: "dbd46c9b-8c60-44d3-a846-788ff0d0607a"). InnerVolumeSpecName "kube-api-access-n74c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 11:59:44.864124 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.864038    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd46c9b-8c60-44d3-a846-788ff0d0607a-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:44.864124 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.864069    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf56x\" (UniqueName: \"kubernetes.io/projected/e8fdbcad-af38-43b9-bc5e-44bf080c472a-kube-api-access-rf56x\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:44.864124 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.864078    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8fdbcad-af38-43b9-bc5e-44bf080c472a-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:44.864124 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.864088    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n74c6\" (UniqueName: \"kubernetes.io/projected/dbd46c9b-8c60-44d3-a846-788ff0d0607a-kube-api-access-n74c6\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:44.973939 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.973910    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"]
Sep 15 11:59:44.976338 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.976314    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2023d5-d75f87b7-s7gcn"]
Sep 15 11:59:44.992946 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.992920    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"]
Sep 15 11:59:44.998176 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:44.998155    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2023d5-7cbc4496fd-bl78q"]
Sep 15 11:59:46.665968 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:46.665935    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd46c9b-8c60-44d3-a846-788ff0d0607a" path="/var/lib/kubelet/pods/dbd46c9b-8c60-44d3-a846-788ff0d0607a/volumes"
Sep 15 11:59:46.666326 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:46.666279    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fdbcad-af38-43b9-bc5e-44bf080c472a" path="/var/lib/kubelet/pods/e8fdbcad-af38-43b9-bc5e-44bf080c472a/volumes"
Sep 15 11:59:51.119656 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.119616    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"]
Sep 15 11:59:51.120201 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120181    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428adb06-e9c7-4747-9561-b55f7931c6aa" containerName="revoke-keys"
Sep 15 11:59:51.120281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120204    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="428adb06-e9c7-4747-9561-b55f7931c6aa" containerName="revoke-keys"
Sep 15 11:59:51.120281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120236    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fdbcad-af38-43b9-bc5e-44bf080c472a" containerName="payload-processing"
Sep 15 11:59:51.120281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120245    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fdbcad-af38-43b9-bc5e-44bf080c472a" containerName="payload-processing"
Sep 15 11:59:51.120281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120259    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd46c9b-8c60-44d3-a846-788ff0d0607a" containerName="payload-pre-processing"
Sep 15 11:59:51.120281 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120268    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd46c9b-8c60-44d3-a846-788ff0d0607a" containerName="payload-pre-processing"
Sep 15 11:59:51.120561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120372    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd46c9b-8c60-44d3-a846-788ff0d0607a" containerName="payload-pre-processing"
Sep 15 11:59:51.120561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120386    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="428adb06-e9c7-4747-9561-b55f7931c6aa" containerName="revoke-keys"
Sep 15 11:59:51.120561 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.120396    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8fdbcad-af38-43b9-bc5e-44bf080c472a" containerName="payload-processing"
Sep 15 11:59:51.125040 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.125020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:51.127733 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.127700    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 11:59:51.127854 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.127742    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 11:59:51.129749 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.129723    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"]
Sep 15 11:59:51.219392 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.219354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfd186f-0919-40d5-be75-f5df0ff41a8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24\" (UID: \"fdfd186f-0919-40d5-be75-f5df0ff41a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:51.320171 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.320123    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfd186f-0919-40d5-be75-f5df0ff41a8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24\" (UID: \"fdfd186f-0919-40d5-be75-f5df0ff41a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:51.320789 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.320767    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfd186f-0919-40d5-be75-f5df0ff41a8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24\" (UID: \"fdfd186f-0919-40d5-be75-f5df0ff41a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:51.436352 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.436321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:51.705632 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:51.705609    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"]
Sep 15 11:59:51.707789 ip-10-0-142-54 kubenswrapper[2574]: W0915 11:59:51.707764    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfd186f_0919_40d5_be75_f5df0ff41a8c.slice/crio-09913771ebe44f686b6fb59ba30c5d524ad1814834b9a02c5a8291fb874b00a2 WatchSource:0}: Error finding container 09913771ebe44f686b6fb59ba30c5d524ad1814834b9a02c5a8291fb874b00a2: Status 404 returned error can't find the container with id 09913771ebe44f686b6fb59ba30c5d524ad1814834b9a02c5a8291fb874b00a2
Sep 15 11:59:52.705064 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:52.704963    2574 generic.go:356] "Generic (PLEG): container finished" podID="fdfd186f-0919-40d5-be75-f5df0ff41a8c" containerID="987425cde9b49acaefaff54e411c63206dacb13e1669d4f0daec3fc9dec85b92" exitCode=0
Sep 15 11:59:52.705064 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:52.705028    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24" event={"ID":"fdfd186f-0919-40d5-be75-f5df0ff41a8c","Type":"ContainerDied","Data":"987425cde9b49acaefaff54e411c63206dacb13e1669d4f0daec3fc9dec85b92"}
Sep 15 11:59:52.705595 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:52.705096    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24" event={"ID":"fdfd186f-0919-40d5-be75-f5df0ff41a8c","Type":"ContainerStarted","Data":"09913771ebe44f686b6fb59ba30c5d524ad1814834b9a02c5a8291fb874b00a2"}
Sep 15 11:59:53.839255 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:53.839231    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:53.943246 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:53.943214    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfd186f-0919-40d5-be75-f5df0ff41a8c-maas-api-service-ca\") pod \"fdfd186f-0919-40d5-be75-f5df0ff41a8c\" (UID: \"fdfd186f-0919-40d5-be75-f5df0ff41a8c\") "
Sep 15 11:59:53.943597 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:53.943576    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfd186f-0919-40d5-be75-f5df0ff41a8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdfd186f-0919-40d5-be75-f5df0ff41a8c" (UID: "fdfd186f-0919-40d5-be75-f5df0ff41a8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 11:59:54.044634 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:54.044559    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdfd186f-0919-40d5-be75-f5df0ff41a8c-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 11:59:54.717079 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:54.717049    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"
Sep 15 11:59:54.717238 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:54.717079    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24" event={"ID":"fdfd186f-0919-40d5-be75-f5df0ff41a8c","Type":"ContainerDied","Data":"09913771ebe44f686b6fb59ba30c5d524ad1814834b9a02c5a8291fb874b00a2"}
Sep 15 11:59:54.717238 ip-10-0-142-54 kubenswrapper[2574]: I0915 11:59:54.717128    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09913771ebe44f686b6fb59ba30c5d524ad1814834b9a02c5a8291fb874b00a2"
Sep 15 12:00:00.137435 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.137392    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"]
Sep 15 12:00:00.137927 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.137903    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdfd186f-0919-40d5-be75-f5df0ff41a8c" containerName="revoke-keys"
Sep 15 12:00:00.137927 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.137918    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdfd186f-0919-40d5-be75-f5df0ff41a8c" containerName="revoke-keys"
Sep 15 12:00:00.138053 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.137987    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdfd186f-0919-40d5-be75-f5df0ff41a8c" containerName="revoke-keys"
Sep 15 12:00:00.141157 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.141134    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"]
Sep 15 12:00:00.141278 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.141253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"
Sep 15 12:00:00.143341 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.143313    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 12:00:00.145062 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.145042    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"
Sep 15 12:00:00.148287 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.148260    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"]
Sep 15 12:00:00.155191 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.154995    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"
Sep 15 12:00:00.164056 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.161914    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"]
Sep 15 12:00:00.170703 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.170679    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"
Sep 15 12:00:00.443610 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.443584    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"]
Sep 15 12:00:00.446095 ip-10-0-142-54 kubenswrapper[2574]: W0915 12:00:00.446065    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de3c65e_7994_48f9_b8c0_22e97fc30a02.slice/crio-36d17949c998535a0106a77112eeb9017b39909166b5f1b00e97bb4309837809 WatchSource:0}: Error finding container 36d17949c998535a0106a77112eeb9017b39909166b5f1b00e97bb4309837809: Status 404 returned error can't find the container with id 36d17949c998535a0106a77112eeb9017b39909166b5f1b00e97bb4309837809
Sep 15 12:00:00.453654 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.453439    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"]
Sep 15 12:00:00.456447 ip-10-0-142-54 kubenswrapper[2574]: W0915 12:00:00.456422    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de52958_6639_4e50_b262_90550e301ee3.slice/crio-ea4339dda0896c66e32a7b93f2dc8e4f6d64ac6eb271c3561e03f8b44aeda4d6 WatchSource:0}: Error finding container ea4339dda0896c66e32a7b93f2dc8e4f6d64ac6eb271c3561e03f8b44aeda4d6: Status 404 returned error can't find the container with id ea4339dda0896c66e32a7b93f2dc8e4f6d64ac6eb271c3561e03f8b44aeda4d6
Sep 15 12:00:00.748657 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.748620    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh" event={"ID":"8de52958-6639-4e50-b262-90550e301ee3","Type":"ContainerStarted","Data":"38323fa0efb05e0cf3fade217af4109a2c545ad9f42e8f9c0e399e7f0c9d14f6"}
Sep 15 12:00:00.748859 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.748684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh" event={"ID":"8de52958-6639-4e50-b262-90550e301ee3","Type":"ContainerStarted","Data":"ea4339dda0896c66e32a7b93f2dc8e4f6d64ac6eb271c3561e03f8b44aeda4d6"}
Sep 15 12:00:00.750193 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.750164    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg" event={"ID":"5de3c65e-7994-48f9-b8c0-22e97fc30a02","Type":"ContainerStarted","Data":"8a69fdea676e61e45ea8f822688d5f314ed58d820647d42cf2dba9f69c0feba7"}
Sep 15 12:00:00.750323 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.750210    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg" event={"ID":"5de3c65e-7994-48f9-b8c0-22e97fc30a02","Type":"ContainerStarted","Data":"36d17949c998535a0106a77112eeb9017b39909166b5f1b00e97bb4309837809"}
Sep 15 12:00:00.762211 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.762090    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh" podStartSLOduration=0.762071163 podStartE2EDuration="762.071163ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:00.761572491 +0000 UTC m=+2094.539248750" watchObservedRunningTime="2026-09-15 12:00:00.762071163 +0000 UTC m=+2094.539747422"
Sep 15 12:00:00.778324 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:00.778269    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg" podStartSLOduration=0.778251015 podStartE2EDuration="778.251015ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:00.777104941 +0000 UTC m=+2094.554781200" watchObservedRunningTime="2026-09-15 12:00:00.778251015 +0000 UTC m=+2094.555927272"
Sep 15 12:00:01.696733 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.696700    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"]
Sep 15 12:00:01.713659 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.713620    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"]
Sep 15 12:00:01.713901 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.713874    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw" podUID="353a8618-d556-405c-8172-0c718a1d564d" containerName="payload-processing" containerID="cri-o://37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b" gracePeriod=30
Sep 15 12:00:01.720853 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.720823    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"]
Sep 15 12:00:01.721073 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.721045    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n" podUID="98770b1b-39e6-4466-a789-8f239f84be38" containerName="payload-pre-processing" containerID="cri-o://cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534" gracePeriod=30
Sep 15 12:00:01.757607 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.757574    2574 generic.go:356] "Generic (PLEG): container finished" podID="5de3c65e-7994-48f9-b8c0-22e97fc30a02" containerID="8a69fdea676e61e45ea8f822688d5f314ed58d820647d42cf2dba9f69c0feba7" exitCode=0
Sep 15 12:00:01.758170 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.757668    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg" event={"ID":"5de3c65e-7994-48f9-b8c0-22e97fc30a02","Type":"ContainerDied","Data":"8a69fdea676e61e45ea8f822688d5f314ed58d820647d42cf2dba9f69c0feba7"}
Sep 15 12:00:01.759581 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.759555    2574 generic.go:356] "Generic (PLEG): container finished" podID="8de52958-6639-4e50-b262-90550e301ee3" containerID="38323fa0efb05e0cf3fade217af4109a2c545ad9f42e8f9c0e399e7f0c9d14f6" exitCode=0
Sep 15 12:00:01.759697 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.759601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh" event={"ID":"8de52958-6639-4e50-b262-90550e301ee3","Type":"ContainerDied","Data":"38323fa0efb05e0cf3fade217af4109a2c545ad9f42e8f9c0e399e7f0c9d14f6"}
Sep 15 12:00:01.872765 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.872730    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"]
Sep 15 12:00:01.876876 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.876845    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-26f1b0-7jp24"]
Sep 15 12:00:01.970167 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:01.970139    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 12:00:02.016681 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.016648    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 12:00:02.024774 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.024743    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmqd9\" (UniqueName: \"kubernetes.io/projected/353a8618-d556-405c-8172-0c718a1d564d-kube-api-access-cmqd9\") pod \"353a8618-d556-405c-8172-0c718a1d564d\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") "
Sep 15 12:00:02.024981 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.024897    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a8618-d556-405c-8172-0c718a1d564d-plugins-config-volume\") pod \"353a8618-d556-405c-8172-0c718a1d564d\" (UID: \"353a8618-d556-405c-8172-0c718a1d564d\") "
Sep 15 12:00:02.025295 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.025272    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353a8618-d556-405c-8172-0c718a1d564d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "353a8618-d556-405c-8172-0c718a1d564d" (UID: "353a8618-d556-405c-8172-0c718a1d564d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:02.026727 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.026708    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353a8618-d556-405c-8172-0c718a1d564d-kube-api-access-cmqd9" (OuterVolumeSpecName: "kube-api-access-cmqd9") pod "353a8618-d556-405c-8172-0c718a1d564d" (UID: "353a8618-d556-405c-8172-0c718a1d564d"). InnerVolumeSpecName "kube-api-access-cmqd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:02.125857 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.125820    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ths9\" (UniqueName: \"kubernetes.io/projected/98770b1b-39e6-4466-a789-8f239f84be38-kube-api-access-5ths9\") pod \"98770b1b-39e6-4466-a789-8f239f84be38\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") "
Sep 15 12:00:02.126113 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.125973    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98770b1b-39e6-4466-a789-8f239f84be38-plugins-config-volume\") pod \"98770b1b-39e6-4466-a789-8f239f84be38\" (UID: \"98770b1b-39e6-4466-a789-8f239f84be38\") "
Sep 15 12:00:02.126228 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.126209    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmqd9\" (UniqueName: \"kubernetes.io/projected/353a8618-d556-405c-8172-0c718a1d564d-kube-api-access-cmqd9\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:02.126287 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.126234    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/353a8618-d556-405c-8172-0c718a1d564d-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:02.126399 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.126381    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98770b1b-39e6-4466-a789-8f239f84be38-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "98770b1b-39e6-4466-a789-8f239f84be38" (UID: "98770b1b-39e6-4466-a789-8f239f84be38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:02.127998 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.127968    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98770b1b-39e6-4466-a789-8f239f84be38-kube-api-access-5ths9" (OuterVolumeSpecName: "kube-api-access-5ths9") pod "98770b1b-39e6-4466-a789-8f239f84be38" (UID: "98770b1b-39e6-4466-a789-8f239f84be38"). InnerVolumeSpecName "kube-api-access-5ths9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:02.227446 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.227355    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ths9\" (UniqueName: \"kubernetes.io/projected/98770b1b-39e6-4466-a789-8f239f84be38-kube-api-access-5ths9\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:02.227446 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.227388    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98770b1b-39e6-4466-a789-8f239f84be38-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:02.666334 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.666299    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdfd186f-0919-40d5-be75-f5df0ff41a8c" path="/var/lib/kubelet/pods/fdfd186f-0919-40d5-be75-f5df0ff41a8c/volumes"
Sep 15 12:00:02.766154 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.766118    2574 generic.go:356] "Generic (PLEG): container finished" podID="353a8618-d556-405c-8172-0c718a1d564d" containerID="37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b" exitCode=0
Sep 15 12:00:02.766592 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.766164    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"
Sep 15 12:00:02.766592 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.766203    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw" event={"ID":"353a8618-d556-405c-8172-0c718a1d564d","Type":"ContainerDied","Data":"37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b"}
Sep 15 12:00:02.766592 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.766252    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw" event={"ID":"353a8618-d556-405c-8172-0c718a1d564d","Type":"ContainerDied","Data":"42a169410864c738d28726a11d532534e4606a0a4a2a314a67ca4b99bf017015"}
Sep 15 12:00:02.766592 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.766280    2574 scope.go:117] "RemoveContainer" containerID="37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b"
Sep 15 12:00:02.767772 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.767713    2574 generic.go:356] "Generic (PLEG): container finished" podID="98770b1b-39e6-4466-a789-8f239f84be38" containerID="cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534" exitCode=0
Sep 15 12:00:02.767772 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.767756    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"
Sep 15 12:00:02.767902 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.767795    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n" event={"ID":"98770b1b-39e6-4466-a789-8f239f84be38","Type":"ContainerDied","Data":"cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534"}
Sep 15 12:00:02.767902 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.767838    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n" event={"ID":"98770b1b-39e6-4466-a789-8f239f84be38","Type":"ContainerDied","Data":"93ca83e49d2352fa447c66046da641aa56253aa6b26d63c7db8244aa2ee0c9fa"}
Sep 15 12:00:02.780092 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.780062    2574 scope.go:117] "RemoveContainer" containerID="37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b"
Sep 15 12:00:02.780752 ip-10-0-142-54 kubenswrapper[2574]: E0915 12:00:02.780722    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b\": container with ID starting with 37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b not found: ID does not exist" containerID="37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b"
Sep 15 12:00:02.780884 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.780765    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b"} err="failed to get container status \"37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b\": rpc error: code = NotFound desc = could not find container \"37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b\": container with ID starting with 37e8f14b8eec65506ae6365cca6179d6758c16531078775818a1c1a3f77ad94b not found: ID does not exist"
Sep 15 12:00:02.780884 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.780789    2574 scope.go:117] "RemoveContainer" containerID="cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534"
Sep 15 12:00:02.792819 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.792772    2574 scope.go:117] "RemoveContainer" containerID="cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534"
Sep 15 12:00:02.793195 ip-10-0-142-54 kubenswrapper[2574]: E0915 12:00:02.793173    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534\": container with ID starting with cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534 not found: ID does not exist" containerID="cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534"
Sep 15 12:00:02.793309 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.793207    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534"} err="failed to get container status \"cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534\": rpc error: code = NotFound desc = could not find container \"cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534\": container with ID starting with cff17ab2124ca3fc25320e40c3b38b52dfea58d0e77f8cbb61f543e1e1fb9534 not found: ID does not exist"
Sep 15 12:00:02.803521 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.802361    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"]
Sep 15 12:00:02.806310 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.805946    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-26f1b0-79ddcfb5c-sqrzw"]
Sep 15 12:00:02.820973 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.815848    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"]
Sep 15 12:00:02.823425 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.823396    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-26f1b0-76f8c99946-qrm8n"]
Sep 15 12:00:02.948965 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.948934    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"
Sep 15 12:00:02.950961 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:02.950936    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"
Sep 15 12:00:03.773856 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.773822    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"
Sep 15 12:00:03.773856 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.773837    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh" event={"ID":"8de52958-6639-4e50-b262-90550e301ee3","Type":"ContainerDied","Data":"ea4339dda0896c66e32a7b93f2dc8e4f6d64ac6eb271c3561e03f8b44aeda4d6"}
Sep 15 12:00:03.774313 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.773882    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4339dda0896c66e32a7b93f2dc8e4f6d64ac6eb271c3561e03f8b44aeda4d6"
Sep 15 12:00:03.776824 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.776801    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"
Sep 15 12:00:03.776969 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.776845    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg" event={"ID":"5de3c65e-7994-48f9-b8c0-22e97fc30a02","Type":"ContainerDied","Data":"36d17949c998535a0106a77112eeb9017b39909166b5f1b00e97bb4309837809"}
Sep 15 12:00:03.776969 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.776906    2574 scope.go:117] "RemoveContainer" containerID="8a69fdea676e61e45ea8f822688d5f314ed58d820647d42cf2dba9f69c0feba7"
Sep 15 12:00:03.801293 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.801265    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"]
Sep 15 12:00:03.804835 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:03.804809    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-26f1b0-29824560-jnzlg"]
Sep 15 12:00:04.666010 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:04.665975    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="353a8618-d556-405c-8172-0c718a1d564d" path="/var/lib/kubelet/pods/353a8618-d556-405c-8172-0c718a1d564d/volumes"
Sep 15 12:00:04.666273 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:04.666259    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5de3c65e-7994-48f9-b8c0-22e97fc30a02" path="/var/lib/kubelet/pods/5de3c65e-7994-48f9-b8c0-22e97fc30a02/volumes"
Sep 15 12:00:04.666564 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:04.666552    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98770b1b-39e6-4466-a789-8f239f84be38" path="/var/lib/kubelet/pods/98770b1b-39e6-4466-a789-8f239f84be38/volumes"
Sep 15 12:00:06.801745 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:06.801718    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 12:00:06.805688 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:06.805666    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 12:00:06.806451 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:06.806432    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 12:00:06.810558 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:06.810535    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 12:00:08.675157 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675121    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"]
Sep 15 12:00:08.675548 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675510    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="353a8618-d556-405c-8172-0c718a1d564d" containerName="payload-processing"
Sep 15 12:00:08.675548 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675522    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="353a8618-d556-405c-8172-0c718a1d564d" containerName="payload-processing"
Sep 15 12:00:08.675548 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675530    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5de3c65e-7994-48f9-b8c0-22e97fc30a02" containerName="cleanup"
Sep 15 12:00:08.675548 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675535    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5de3c65e-7994-48f9-b8c0-22e97fc30a02" containerName="cleanup"
Sep 15 12:00:08.675548 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675545    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98770b1b-39e6-4466-a789-8f239f84be38" containerName="payload-pre-processing"
Sep 15 12:00:08.675548 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675551    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="98770b1b-39e6-4466-a789-8f239f84be38" containerName="payload-pre-processing"
Sep 15 12:00:08.675770 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675559    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de52958-6639-4e50-b262-90550e301ee3" containerName="cleanup"
Sep 15 12:00:08.675770 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675564    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de52958-6639-4e50-b262-90550e301ee3" containerName="cleanup"
Sep 15 12:00:08.675770 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675622    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="98770b1b-39e6-4466-a789-8f239f84be38" containerName="payload-pre-processing"
Sep 15 12:00:08.675770 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675633    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de52958-6639-4e50-b262-90550e301ee3" containerName="cleanup"
Sep 15 12:00:08.675770 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675639    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="353a8618-d556-405c-8172-0c718a1d564d" containerName="payload-processing"
Sep 15 12:00:08.675770 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.675644    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5de3c65e-7994-48f9-b8c0-22e97fc30a02" containerName="cleanup"
Sep 15 12:00:08.680220 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.680201    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:08.682305 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.682283    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 12:00:08.682456 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.682440    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2w6g9\""
Sep 15 12:00:08.686232 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.686208    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"]
Sep 15 12:00:08.790614 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.790570    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68510ab-3ba3-44be-bc02-9dbce9731f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz\" (UID: \"a68510ab-3ba3-44be-bc02-9dbce9731f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:08.891367 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.891332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68510ab-3ba3-44be-bc02-9dbce9731f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz\" (UID: \"a68510ab-3ba3-44be-bc02-9dbce9731f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:08.891991 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.891966    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68510ab-3ba3-44be-bc02-9dbce9731f9f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz\" (UID: \"a68510ab-3ba3-44be-bc02-9dbce9731f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:08.991652 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:08.991570    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:09.260115 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:09.259932    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"]
Sep 15 12:00:09.263013 ip-10-0-142-54 kubenswrapper[2574]: W0915 12:00:09.262987    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68510ab_3ba3_44be_bc02_9dbce9731f9f.slice/crio-1e15b328def1220e3f600ca44d0adf49d31db2c0aabb6b4029a5fba28a56cebb WatchSource:0}: Error finding container 1e15b328def1220e3f600ca44d0adf49d31db2c0aabb6b4029a5fba28a56cebb: Status 404 returned error can't find the container with id 1e15b328def1220e3f600ca44d0adf49d31db2c0aabb6b4029a5fba28a56cebb
Sep 15 12:00:09.810026 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:09.809993    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz" event={"ID":"a68510ab-3ba3-44be-bc02-9dbce9731f9f","Type":"ContainerStarted","Data":"4c59f66e7630dbcc2ab3f14ffb6c7b7a39ac535d4653a3ec2e7e2758cde9695a"}
Sep 15 12:00:09.810482 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:09.810050    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz" event={"ID":"a68510ab-3ba3-44be-bc02-9dbce9731f9f","Type":"ContainerStarted","Data":"1e15b328def1220e3f600ca44d0adf49d31db2c0aabb6b4029a5fba28a56cebb"}
Sep 15 12:00:09.825684 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:09.825633    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz" podStartSLOduration=1.825615939 podStartE2EDuration="1.825615939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:00:09.824572228 +0000 UTC m=+2103.602248498" watchObservedRunningTime="2026-09-15 12:00:09.825615939 +0000 UTC m=+2103.603292226"
Sep 15 12:00:10.817545 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:10.817508    2574 generic.go:356] "Generic (PLEG): container finished" podID="a68510ab-3ba3-44be-bc02-9dbce9731f9f" containerID="4c59f66e7630dbcc2ab3f14ffb6c7b7a39ac535d4653a3ec2e7e2758cde9695a" exitCode=0
Sep 15 12:00:10.817918 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:10.817596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz" event={"ID":"a68510ab-3ba3-44be-bc02-9dbce9731f9f","Type":"ContainerDied","Data":"4c59f66e7630dbcc2ab3f14ffb6c7b7a39ac535d4653a3ec2e7e2758cde9695a"}
Sep 15 12:00:11.955623 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:11.955597    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:12.121561 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:12.121424    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68510ab-3ba3-44be-bc02-9dbce9731f9f-maas-api-service-ca\") pod \"a68510ab-3ba3-44be-bc02-9dbce9731f9f\" (UID: \"a68510ab-3ba3-44be-bc02-9dbce9731f9f\") "
Sep 15 12:00:12.121878 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:12.121855    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68510ab-3ba3-44be-bc02-9dbce9731f9f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a68510ab-3ba3-44be-bc02-9dbce9731f9f" (UID: "a68510ab-3ba3-44be-bc02-9dbce9731f9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:12.222661 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:12.222603    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a68510ab-3ba3-44be-bc02-9dbce9731f9f-maas-api-service-ca\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:12.829712 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:12.829680    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"
Sep 15 12:00:12.829888 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:12.829715    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz" event={"ID":"a68510ab-3ba3-44be-bc02-9dbce9731f9f","Type":"ContainerDied","Data":"1e15b328def1220e3f600ca44d0adf49d31db2c0aabb6b4029a5fba28a56cebb"}
Sep 15 12:00:12.829888 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:12.829763    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e15b328def1220e3f600ca44d0adf49d31db2c0aabb6b4029a5fba28a56cebb"
Sep 15 12:00:19.427779 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.427743    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"]
Sep 15 12:00:19.434772 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.434739    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-dd6959-29824560-46mrh"]
Sep 15 12:00:19.466212 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.466169    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"]
Sep 15 12:00:19.466452 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.466425    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2" podUID="dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" containerName="payload-pre-processing" containerID="cri-o://487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050" gracePeriod=30
Sep 15 12:00:19.623861 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.623828    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"]
Sep 15 12:00:19.627573 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.627548    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-dd6959-vw2cz"]
Sep 15 12:00:19.721009 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.720986    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 12:00:19.866782 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.866744    2574 generic.go:356] "Generic (PLEG): container finished" podID="dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" containerID="487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050" exitCode=0
Sep 15 12:00:19.866963 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.866789    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"
Sep 15 12:00:19.866963 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.866833    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2" event={"ID":"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd","Type":"ContainerDied","Data":"487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050"}
Sep 15 12:00:19.866963 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.866885    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2" event={"ID":"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd","Type":"ContainerDied","Data":"a1143c329d8aaf5c6a9b33a8ce82a8e9a8bb72c33e030a900d54176c84d444eb"}
Sep 15 12:00:19.866963 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.866908    2574 scope.go:117] "RemoveContainer" containerID="487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050"
Sep 15 12:00:19.875538 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.875521    2574 scope.go:117] "RemoveContainer" containerID="487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050"
Sep 15 12:00:19.875798 ip-10-0-142-54 kubenswrapper[2574]: E0915 12:00:19.875780    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050\": container with ID starting with 487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050 not found: ID does not exist" containerID="487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050"
Sep 15 12:00:19.875848 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.875815    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050"} err="failed to get container status \"487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050\": rpc error: code = NotFound desc = could not find container \"487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050\": container with ID starting with 487570c15683c2d8410e37879d885d90f4bfdf69264664950d2788cd99ca3050 not found: ID does not exist"
Sep 15 12:00:19.901963 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.901935    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmcsr\" (UniqueName: \"kubernetes.io/projected/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-kube-api-access-dmcsr\") pod \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") "
Sep 15 12:00:19.902095 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.901972    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-plugins-config-volume\") pod \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\" (UID: \"dacbe108-cc0d-4108-ad49-cbcf39bfc9cd\") "
Sep 15 12:00:19.902388 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.902364    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" (UID: "dacbe108-cc0d-4108-ad49-cbcf39bfc9cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 12:00:19.903837 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:19.903819    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-kube-api-access-dmcsr" (OuterVolumeSpecName: "kube-api-access-dmcsr") pod "dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" (UID: "dacbe108-cc0d-4108-ad49-cbcf39bfc9cd"). InnerVolumeSpecName "kube-api-access-dmcsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:00:20.003415 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.003334    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmcsr\" (UniqueName: \"kubernetes.io/projected/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-kube-api-access-dmcsr\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:20.003415 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.003361    2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd-plugins-config-volume\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:00:20.187617 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.187576    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"]
Sep 15 12:00:20.190957 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.190927    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-dd6959-5c59f85b94-76jg2"]
Sep 15 12:00:20.665761 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.665727    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de52958-6639-4e50-b262-90550e301ee3" path="/var/lib/kubelet/pods/8de52958-6639-4e50-b262-90550e301ee3/volumes"
Sep 15 12:00:20.666132 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.666068    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a68510ab-3ba3-44be-bc02-9dbce9731f9f" path="/var/lib/kubelet/pods/a68510ab-3ba3-44be-bc02-9dbce9731f9f/volumes"
Sep 15 12:00:20.666424 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:00:20.666410    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" path="/var/lib/kubelet/pods/dacbe108-cc0d-4108-ad49-cbcf39bfc9cd/volumes"
Sep 15 12:04:56.349239 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349183    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"]
Sep 15 12:04:56.349837 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349817    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" containerName="payload-pre-processing"
Sep 15 12:04:56.349928 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349839    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" containerName="payload-pre-processing"
Sep 15 12:04:56.349928 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349865    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a68510ab-3ba3-44be-bc02-9dbce9731f9f" containerName="revoke-keys"
Sep 15 12:04:56.349928 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349873    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a68510ab-3ba3-44be-bc02-9dbce9731f9f" containerName="revoke-keys"
Sep 15 12:04:56.350086 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349978    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacbe108-cc0d-4108-ad49-cbcf39bfc9cd" containerName="payload-pre-processing"
Sep 15 12:04:56.350086 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.349994    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a68510ab-3ba3-44be-bc02-9dbce9731f9f" containerName="revoke-keys"
Sep 15 12:04:56.353154 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.353131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.355200 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.355177    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9b1f7de3-kserve-self-signed-certs\""
Sep 15 12:04:56.362025 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.362003    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"]
Sep 15 12:04:56.470386 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.470353    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwqrg\" (UniqueName: \"kubernetes.io/projected/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-kube-api-access-wwqrg\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.470558 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.470405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-dshm\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.470558 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.470429    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-home\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.470644 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.470547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tmp-dir\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.470644 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.470582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-model-cache\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.470644 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.470615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tls-certs\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.571681 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.571643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-model-cache\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.571681 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.571687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tls-certs\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.571944 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.571737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwqrg\" (UniqueName: \"kubernetes.io/projected/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-kube-api-access-wwqrg\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.571944 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.571768    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-dshm\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.571944 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.571788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-home\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.572098 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.571955    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tmp-dir\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.572140 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.572119    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-model-cache\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.572195 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.572158    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-home\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.572300 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.572280    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tmp-dir\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.574088 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.574066    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-dshm\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.574263 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.574246    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tls-certs\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.579205 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.579180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwqrg\" (UniqueName: \"kubernetes.io/projected/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-kube-api-access-wwqrg\") pod \"e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") " pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.662801 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.662775    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:04:56.935255 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:56.935225    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"]
Sep 15 12:04:56.938003 ip-10-0-142-54 kubenswrapper[2574]: W0915 12:04:56.937973    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod224fc1c4_07fb_41d0_943d_fd569ef1ffd1.slice/crio-ded620cc2a4539ce0b52ec6645d04723aee557237302b458b3a0496fe2931e37 WatchSource:0}: Error finding container ded620cc2a4539ce0b52ec6645d04723aee557237302b458b3a0496fe2931e37: Status 404 returned error can't find the container with id ded620cc2a4539ce0b52ec6645d04723aee557237302b458b3a0496fe2931e37
Sep 15 12:04:57.327086 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:57.326999    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc" event={"ID":"224fc1c4-07fb-41d0-943d-fd569ef1ffd1","Type":"ContainerStarted","Data":"4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b"}
Sep 15 12:04:57.327086 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:57.327041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc" event={"ID":"224fc1c4-07fb-41d0-943d-fd569ef1ffd1","Type":"ContainerStarted","Data":"ded620cc2a4539ce0b52ec6645d04723aee557237302b458b3a0496fe2931e37"}
Sep 15 12:04:57.343057 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:04:57.343003    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc" podStartSLOduration=1.342987125 podStartE2EDuration="1.342987125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 12:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:04:57.34215879 +0000 UTC m=+2391.119835046" watchObservedRunningTime="2026-09-15 12:04:57.342987125 +0000 UTC m=+2391.120663382"
Sep 15 12:05:06.666680 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.666649    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:05:06.666680 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.666688    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:05:06.676358 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.676332    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:05:06.834573 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.834546    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 12:05:06.845460 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.845428    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 12:05:06.849719 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.849698    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 12:05:06.853778 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:06.853758    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 12:05:07.200185 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:07.200156    2574 scope.go:117] "RemoveContainer" containerID="16e895691f1a15c5c68994c6e3e8098e98b4c3ca89adddfceda03a7cabff75e6"
Sep 15 12:05:07.208517 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:07.208482    2574 scope.go:117] "RemoveContainer" containerID="071211fe051c6d3119013b99994620dfa94e3d267773d158da00d1d8ddb5bfd5"
Sep 15 12:05:07.393171 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:07.393145    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:05:26.000463 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:26.000427    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"]
Sep 15 12:05:41.021035 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.020950    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc" podUID="224fc1c4-07fb-41d0-943d-fd569ef1ffd1" containerName="main" containerID="cri-o://4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b" gracePeriod=45
Sep 15 12:05:41.265963 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.265937    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:05:41.405408 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405378    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-model-cache\") pod \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") "
Sep 15 12:05:41.405605 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405490    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tls-certs\") pod \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") "
Sep 15 12:05:41.405605 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405540    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwqrg\" (UniqueName: \"kubernetes.io/projected/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-kube-api-access-wwqrg\") pod \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") "
Sep 15 12:05:41.405605 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tmp-dir\") pod \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") "
Sep 15 12:05:41.405605 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405592    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-dshm\") pod \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") "
Sep 15 12:05:41.405605 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405608    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-home\") pod \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\" (UID: \"224fc1c4-07fb-41d0-943d-fd569ef1ffd1\") "
Sep 15 12:05:41.405880 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405692    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-model-cache" (OuterVolumeSpecName: "model-cache") pod "224fc1c4-07fb-41d0-943d-fd569ef1ffd1" (UID: "224fc1c4-07fb-41d0-943d-fd569ef1ffd1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:05:41.405941 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405875    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "224fc1c4-07fb-41d0-943d-fd569ef1ffd1" (UID: "224fc1c4-07fb-41d0-943d-fd569ef1ffd1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:05:41.405941 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405913    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-home" (OuterVolumeSpecName: "home") pod "224fc1c4-07fb-41d0-943d-fd569ef1ffd1" (UID: "224fc1c4-07fb-41d0-943d-fd569ef1ffd1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:05:41.406030 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.405956    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-model-cache\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:05:41.407778 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.407754    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "224fc1c4-07fb-41d0-943d-fd569ef1ffd1" (UID: "224fc1c4-07fb-41d0-943d-fd569ef1ffd1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 12:05:41.407860 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.407781    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-kube-api-access-wwqrg" (OuterVolumeSpecName: "kube-api-access-wwqrg") pod "224fc1c4-07fb-41d0-943d-fd569ef1ffd1" (UID: "224fc1c4-07fb-41d0-943d-fd569ef1ffd1"). InnerVolumeSpecName "kube-api-access-wwqrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 12:05:41.407860 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.407838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-dshm" (OuterVolumeSpecName: "dshm") pod "224fc1c4-07fb-41d0-943d-fd569ef1ffd1" (UID: "224fc1c4-07fb-41d0-943d-fd569ef1ffd1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 12:05:41.507195 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.507150    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tls-certs\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:05:41.507195 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.507189    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwqrg\" (UniqueName: \"kubernetes.io/projected/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-kube-api-access-wwqrg\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:05:41.507195 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.507202    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-tmp-dir\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:05:41.507449 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.507215    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-dshm\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:05:41.507449 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.507226    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/224fc1c4-07fb-41d0-943d-fd569ef1ffd1-home\") on node \"ip-10-0-142-54.ec2.internal\" DevicePath \"\""
Sep 15 12:05:41.557912 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.557880    2574 generic.go:356] "Generic (PLEG): container finished" podID="224fc1c4-07fb-41d0-943d-fd569ef1ffd1" containerID="4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b" exitCode=0
Sep 15 12:05:41.558085 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.557920    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"
Sep 15 12:05:41.558085 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.557972    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc" event={"ID":"224fc1c4-07fb-41d0-943d-fd569ef1ffd1","Type":"ContainerDied","Data":"4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b"}
Sep 15 12:05:41.558085 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.558021    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc" event={"ID":"224fc1c4-07fb-41d0-943d-fd569ef1ffd1","Type":"ContainerDied","Data":"ded620cc2a4539ce0b52ec6645d04723aee557237302b458b3a0496fe2931e37"}
Sep 15 12:05:41.558085 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.558044    2574 scope.go:117] "RemoveContainer" containerID="4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b"
Sep 15 12:05:41.567066 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.567049    2574 scope.go:117] "RemoveContainer" containerID="4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b"
Sep 15 12:05:41.567317 ip-10-0-142-54 kubenswrapper[2574]: E0915 12:05:41.567301    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b\": container with ID starting with 4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b not found: ID does not exist" containerID="4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b"
Sep 15 12:05:41.567369 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.567326    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b"} err="failed to get container status \"4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b\": rpc error: code = NotFound desc = could not find container \"4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b\": container with ID starting with 4a6b3ef3b23636a9482ec3fd864f7c053c263e8a068ddf468bbef6c1afa2b75b not found: ID does not exist"
Sep 15 12:05:41.578182 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.578158    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"]
Sep 15 12:05:41.583637 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:41.583617    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9b1f7de3-kserve-7b575f47b7-gqwkc"]
Sep 15 12:05:42.665015 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:05:42.664983    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224fc1c4-07fb-41d0-943d-fd569ef1ffd1" path="/var/lib/kubelet/pods/224fc1c4-07fb-41d0-943d-fd569ef1ffd1/volumes"
Sep 15 12:06:07.230820 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:07.230711    2574 scope.go:117] "RemoveContainer" containerID="4ba7a52f5a858a924bca6385d93cbddc507621d12416215c7b8b64d10606c58f"
Sep 15 12:06:07.241551 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:07.241530    2574 scope.go:117] "RemoveContainer" containerID="38323fa0efb05e0cf3fade217af4109a2c545ad9f42e8f9c0e399e7f0c9d14f6"
Sep 15 12:06:07.249158 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:07.249143    2574 scope.go:117] "RemoveContainer" containerID="987425cde9b49acaefaff54e411c63206dacb13e1669d4f0daec3fc9dec85b92"
Sep 15 12:06:17.868993 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:17.868955    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64f56c46b4-wz7xb_cc7b13d7-c6a7-43f5-81be-17be7baa5716/authorino/0.log"
Sep 15 12:06:23.598415 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:23.598378    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-99f4658c8-nmn4h_196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f/manager/0.log"
Sep 15 12:06:23.605148 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:23.605123    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-99f4658c8-nmn4h_196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f/copy-manifests/0.log"
Sep 15 12:06:24.381049 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:24.381018    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29824545-867x5_7d9e9e9e-6d33-4f42-b7e8-70a80d71aaca/cleanup/0.log"
Sep 15 12:06:25.351080 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:25.351044    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64f56c46b4-wz7xb_cc7b13d7-c6a7-43f5-81be-17be7baa5716/authorino/0.log"
Sep 15 12:06:26.025029 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:26.024996    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6gpdr_e2da21ec-b8c8-4daf-b9c5-340991424fe1/manager/0.log"
Sep 15 12:06:26.703805 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:26.703763    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59bc95f796-gmfhw_dd03552e-6b42-4d24-a27d-1ce40473cffc/kube-auth-proxy/0.log"
Sep 15 12:06:26.933746 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:26.933718    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkql_ab02359b-112f-4103-bc03-344ca271d9a9/payload-pre-processing/0.log"
Sep 15 12:06:27.163586 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:27.163563    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d486bd584-zz2nb_7123915d-2619-4989-adf4-f56b500cd784/router/0.log"
Sep 15 12:06:27.508019 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:27.507930    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rgdb7_cf424776-9a5e-4b56-8f19-c8135bfc42d8/main/0.log"
Sep 15 12:06:27.854037 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:27.853961    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-p6dtf_d6d671c9-5631-4ccc-b327-34ce9e347094/main/0.log"
Sep 15 12:06:27.963152 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:27.963121    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk6p54_26567ed4-0ae6-46f4-9f02-1aa15d37f41b/main/0.log"
Sep 15 12:06:28.190622 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:28.190597    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-6h9fc_ccf95788-06f9-452c-8236-e1b0bceae601/main/0.log"
Sep 15 12:06:36.514704 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:36.514657    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-75nth_e2c4e4cb-1aed-4d57-a196-f312af6bfe54/global-pull-secret-syncer/0.log"
Sep 15 12:06:36.657553 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:36.657522    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nwhzn_54875eb0-167a-4e37-941a-7cde822004a2/konnectivity-agent/0.log"
Sep 15 12:06:36.723565 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:36.723532    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-54.ec2.internal_839baa2c846d8b905bce654725874020/haproxy/0.log"
Sep 15 12:06:41.194432 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:41.194400    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64f56c46b4-wz7xb_cc7b13d7-c6a7-43f5-81be-17be7baa5716/authorino/0.log"
Sep 15 12:06:41.395413 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:41.395384    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-6gpdr_e2da21ec-b8c8-4daf-b9c5-340991424fe1/manager/0.log"
Sep 15 12:06:43.023082 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.023053    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-595h2_e139383e-c857-4fa8-bad5-2d3599a92756/cluster-monitoring-operator/0.log"
Sep 15 12:06:43.049748 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.049704    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-9vl5v_9801c738-55f4-478e-8885-0bead54cda7e/kube-state-metrics/0.log"
Sep 15 12:06:43.064275 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.064244    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-9vl5v_9801c738-55f4-478e-8885-0bead54cda7e/kube-rbac-proxy-main/0.log"
Sep 15 12:06:43.086110 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.086089    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-9vl5v_9801c738-55f4-478e-8885-0bead54cda7e/kube-rbac-proxy-self/0.log"
Sep 15 12:06:43.378904 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.378876    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjl7v_76e62cfc-5992-4d2f-b7ed-798e336c941c/node-exporter/0.log"
Sep 15 12:06:43.405094 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.405062    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjl7v_76e62cfc-5992-4d2f-b7ed-798e336c941c/kube-rbac-proxy/0.log"
Sep 15 12:06:43.425180 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.425155    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wjl7v_76e62cfc-5992-4d2f-b7ed-798e336c941c/init-textfile/0.log"
Sep 15 12:06:43.472968 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.472942    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dx44p_0e60bc0e-6693-4110-9277-e2af543c7850/kube-rbac-proxy-main/0.log"
Sep 15 12:06:43.496008 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.495965    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dx44p_0e60bc0e-6693-4110-9277-e2af543c7850/kube-rbac-proxy-self/0.log"
Sep 15 12:06:43.516564 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.516535    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-dx44p_0e60bc0e-6693-4110-9277-e2af543c7850/openshift-state-metrics/0.log"
Sep 15 12:06:43.553342 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.553311    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/prometheus/0.log"
Sep 15 12:06:43.571439 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.571418    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/config-reloader/0.log"
Sep 15 12:06:43.592900 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.592880    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/thanos-sidecar/0.log"
Sep 15 12:06:43.617218 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.617164    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/kube-rbac-proxy-web/0.log"
Sep 15 12:06:43.637111 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.637081    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/kube-rbac-proxy/0.log"
Sep 15 12:06:43.656180 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.656158    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/kube-rbac-proxy-thanos/0.log"
Sep 15 12:06:43.673536 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.673518    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d025fcff-e604-47aa-8ea1-11e4bf6451c0/init-config-reloader/0.log"
Sep 15 12:06:43.702489 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.702471    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-t7vf2_ac8d0cdf-af4a-4b1d-89ec-bbe447566941/prometheus-operator/0.log"
Sep 15 12:06:43.725215 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.725190    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-t7vf2_ac8d0cdf-af4a-4b1d-89ec-bbe447566941/kube-rbac-proxy/0.log"
Sep 15 12:06:43.880784 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.880709    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675878854-twdhs_3fe327d1-6caf-4695-972b-e9a1f0448c9a/thanos-query/0.log"
Sep 15 12:06:43.906783 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.906728    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675878854-twdhs_3fe327d1-6caf-4695-972b-e9a1f0448c9a/kube-rbac-proxy-web/0.log"
Sep 15 12:06:43.929429 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.929407    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675878854-twdhs_3fe327d1-6caf-4695-972b-e9a1f0448c9a/kube-rbac-proxy/0.log"
Sep 15 12:06:43.955461 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:43.955430    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675878854-twdhs_3fe327d1-6caf-4695-972b-e9a1f0448c9a/prom-label-proxy/0.log"
Sep 15 12:06:44.015044 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:44.015003    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675878854-twdhs_3fe327d1-6caf-4695-972b-e9a1f0448c9a/kube-rbac-proxy-rules/0.log"
Sep 15 12:06:44.038508 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:44.038474    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6675878854-twdhs_3fe327d1-6caf-4695-972b-e9a1f0448c9a/kube-rbac-proxy-metrics/0.log"
Sep 15 12:06:46.215944 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:46.215914    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/2.log"
Sep 15 12:06:46.220207 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:46.220183    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8czc5_b6982f91-29a9-4be1-aab3-a32da7419a25/console-operator/3.log"
Sep 15 12:06:48.622879 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:48.622851    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r6f5v_63f1b763-8272-482d-aa51-3dad53ae20d0/dns/0.log"
Sep 15 12:06:48.644629 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:48.644607    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-r6f5v_63f1b763-8272-482d-aa51-3dad53ae20d0/kube-rbac-proxy/0.log"
Sep 15 12:06:48.713323 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:48.713293    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jjhhp_285acd19-2f05-4e65-a29e-f44e7c1f84e6/dns-node-resolver/0.log"
Sep 15 12:06:49.264557 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.264531    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gwcfc_6c9f072c-14e7-4808-b8a2-6ee04358d01d/node-ca/0.log"
Sep 15 12:06:49.520852 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.520764    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"]
Sep 15 12:06:49.521246 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.521229    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224fc1c4-07fb-41d0-943d-fd569ef1ffd1" containerName="main"
Sep 15 12:06:49.521340 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.521248    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="224fc1c4-07fb-41d0-943d-fd569ef1ffd1" containerName="main"
Sep 15 12:06:49.521409 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.521395    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="224fc1c4-07fb-41d0-943d-fd569ef1ffd1" containerName="main"
Sep 15 12:06:49.524554 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.524532    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.526542 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.526520    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rmwp6\"/\"default-dockercfg-k4p5p\""
Sep 15 12:06:49.526642 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.526617    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmwp6\"/\"openshift-service-ca.crt\""
Sep 15 12:06:49.527157 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.527140    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rmwp6\"/\"kube-root-ca.crt\""
Sep 15 12:06:49.531050 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.531026    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"]
Sep 15 12:06:49.615074 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.615042    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7fx\" (UniqueName: \"kubernetes.io/projected/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-kube-api-access-px7fx\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.615247 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.615098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-proc\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.615247 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.615151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-podres\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.615247 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.615196    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-sys\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.615247 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.615228    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-lib-modules\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716079 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716030    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7fx\" (UniqueName: \"kubernetes.io/projected/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-kube-api-access-px7fx\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716203    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-proc\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716259    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-podres\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716305    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-proc\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716315    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-sys\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-lib-modules\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-podres\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716449    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-sys\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.716527 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.716488    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-lib-modules\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.724042 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.724010    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7fx\" (UniqueName: \"kubernetes.io/projected/e0aab88a-d36e-49f1-ae63-66abf49a6e0e-kube-api-access-px7fx\") pod \"perf-node-gather-daemonset-cnx45\" (UID: \"e0aab88a-d36e-49f1-ae63-66abf49a6e0e\") " pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:49.834851 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:49.834765    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:50.105647 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.105620    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"]
Sep 15 12:06:50.108092 ip-10-0-142-54 kubenswrapper[2574]: W0915 12:06:50.108061    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0aab88a_d36e_49f1_ae63_66abf49a6e0e.slice/crio-a16385555a5ec11a5294cfd128b87dc71256d4735492950c8b4c781720356b60 WatchSource:0}: Error finding container a16385555a5ec11a5294cfd128b87dc71256d4735492950c8b4c781720356b60: Status 404 returned error can't find the container with id a16385555a5ec11a5294cfd128b87dc71256d4735492950c8b4c781720356b60
Sep 15 12:06:50.731403 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.731374    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59bc95f796-gmfhw_dd03552e-6b42-4d24-a27d-1ce40473cffc/kube-auth-proxy/0.log"
Sep 15 12:06:50.791751 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.791723    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkql_ab02359b-112f-4103-bc03-344ca271d9a9/payload-pre-processing/0.log"
Sep 15 12:06:50.840192 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.840167    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d486bd584-zz2nb_7123915d-2619-4989-adf4-f56b500cd784/router/0.log"
Sep 15 12:06:50.913308 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.913271    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45" event={"ID":"e0aab88a-d36e-49f1-ae63-66abf49a6e0e","Type":"ContainerStarted","Data":"5d43c000e3ddad6d22dc5881720e3ae0210f8c077998a027c59cfc7aea9c6e2a"}
Sep 15 12:06:50.913478 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.913323    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45" event={"ID":"e0aab88a-d36e-49f1-ae63-66abf49a6e0e","Type":"ContainerStarted","Data":"a16385555a5ec11a5294cfd128b87dc71256d4735492950c8b4c781720356b60"}
Sep 15 12:06:50.913557 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.913526    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:50.929362 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:50.929310    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45" podStartSLOduration=1.929294612 podStartE2EDuration="1.929294612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 12:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 12:06:50.928348477 +0000 UTC m=+2504.706024734" watchObservedRunningTime="2026-09-15 12:06:50.929294612 +0000 UTC m=+2504.706970869"
Sep 15 12:06:51.812604 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:51.812574    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pmlm_b0a6f3ad-dee3-4cfe-8d3c-173165b2725a/serve-healthcheck-canary/0.log"
Sep 15 12:06:52.317244 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:52.317218    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-j9rf4_16a89553-0aa8-4f53-9a52-2ed4ae18c722/insights-operator/0.log"
Sep 15 12:06:52.317791 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:52.317773    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-j9rf4_16a89553-0aa8-4f53-9a52-2ed4ae18c722/insights-operator/1.log"
Sep 15 12:06:52.337189 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:52.337169    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hff6_2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36/kube-rbac-proxy/0.log"
Sep 15 12:06:52.355125 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:52.355103    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hff6_2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36/exporter/0.log"
Sep 15 12:06:52.374227 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:52.374205    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9hff6_2e5d0dc1-c426-4c8f-89cc-e4b53f16cc36/extractor/0.log"
Sep 15 12:06:55.193139 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:55.193104    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-99f4658c8-nmn4h_196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f/manager/0.log"
Sep 15 12:06:55.210059 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:55.210033    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-99f4658c8-nmn4h_196ae2e9-86c5-4f2c-9f5a-8db1acf8c44f/copy-manifests/0.log"
Sep 15 12:06:56.474650 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:56.474623    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f6db84f6f-zqzl9_30ba6338-7204-4efb-b2e7-800454edb3ff/manager/0.log"
Sep 15 12:06:56.929561 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:56.929533    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rmwp6/perf-node-gather-daemonset-cnx45"
Sep 15 12:06:57.303921 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:57.303834    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59bc95f796-gmfhw_dd03552e-6b42-4d24-a27d-1ce40473cffc/kube-auth-proxy/0.log"
Sep 15 12:06:57.348946 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:57.348920    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkql_ab02359b-112f-4103-bc03-344ca271d9a9/payload-pre-processing/0.log"
Sep 15 12:06:57.381420 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:57.381387    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d486bd584-zz2nb_7123915d-2619-4989-adf4-f56b500cd784/router/0.log"
Sep 15 12:06:59.569547 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:59.569515    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59bc95f796-gmfhw_dd03552e-6b42-4d24-a27d-1ce40473cffc/kube-auth-proxy/0.log"
Sep 15 12:06:59.620390 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:59.620315    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-6lkql_ab02359b-112f-4103-bc03-344ca271d9a9/payload-pre-processing/0.log"
Sep 15 12:06:59.667730 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:06:59.667696    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-d486bd584-zz2nb_7123915d-2619-4989-adf4-f56b500cd784/router/0.log"
Sep 15 12:07:02.413285 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:02.413257    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9brms_a02c4676-331d-453f-ac98-9e153472814c/kube-storage-version-migrator-operator/1.log"
Sep 15 12:07:02.414220 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:02.414200    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9brms_a02c4676-331d-453f-ac98-9e153472814c/kube-storage-version-migrator-operator/0.log"
Sep 15 12:07:03.554731 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.554703    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/kube-multus-additional-cni-plugins/0.log"
Sep 15 12:07:03.574755 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.574702    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/egress-router-binary-copy/0.log"
Sep 15 12:07:03.594757 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.594732    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/cni-plugins/0.log"
Sep 15 12:07:03.613436 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.613412    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/bond-cni-plugin/0.log"
Sep 15 12:07:03.637407 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.637366    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/routeoverride-cni/0.log"
Sep 15 12:07:03.654871 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.654844    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/whereabouts-cni-bincopy/0.log"
Sep 15 12:07:03.672052 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.672030    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdv4q_acfbfdf0-00c1-4101-a771-797fe7db1917/whereabouts-cni/0.log"
Sep 15 12:07:03.893300 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.893275    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-v6xrw_0eb33362-b615-461f-b096-a6acdee64724/kube-multus/0.log"
Sep 15 12:07:03.968992 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.968962    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4q24x_125ade64-8f23-480a-906c-4400b6463813/network-metrics-daemon/0.log"
Sep 15 12:07:03.984303 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:03.984280    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4q24x_125ade64-8f23-480a-906c-4400b6463813/kube-rbac-proxy/0.log"
Sep 15 12:07:04.930928 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:04.930882    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-controller/0.log"
Sep 15 12:07:04.949882 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:04.949851    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/0.log"
Sep 15 12:07:04.950032 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:04.950005    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovn-acl-logging/1.log"
Sep 15 12:07:04.969718 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:04.969689    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/kube-rbac-proxy-node/0.log"
Sep 15 12:07:04.993404 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:04.993378    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/kube-rbac-proxy-ovn-metrics/0.log"
Sep 15 12:07:05.019054 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:05.019028    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/northd/0.log"
Sep 15 12:07:05.043201 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:05.043171    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/nbdb/0.log"
Sep 15 12:07:05.065197 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:05.065172    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/sbdb/0.log"
Sep 15 12:07:05.124324 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:05.124289    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-89gtx_d15d98ac-db1a-40e7-892a-ae31681023e1/ovnkube-controller/0.log"
Sep 15 12:07:06.623041 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:06.623012    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-6w6m6_feb875f0-4764-40ac-9562-0a624e2a3f73/check-endpoints/0.log"
Sep 15 12:07:06.683351 ip-10-0-142-54 kubenswrapper[2574]: I0915 12:07:06.683323    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fdwrx_86729383-b3ac-4559-b996-929287275e55/network-check-target-container/0.log"