Sep 10 17:09:42.554941 ip-10-0-137-19 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 17:09:42.554951 ip-10-0-137-19 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 17:09:42.554959 ip-10-0-137-19 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 17:09:42.555181 ip-10-0-137-19 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 17:09:52.602830 ip-10-0-137-19 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 17:09:52.602922 ip-10-0-137-19 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 71a12f57ade34e019cfc7e345c33ea27 --
Sep 10 17:11:54.664959 ip-10-0-137-19 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 17:11:55.049614 ip-10-0-137-19 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:11:55.049614 ip-10-0-137-19 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 17:11:55.049614 ip-10-0-137-19 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:11:55.049614 ip-10-0-137-19 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 17:11:55.049614 ip-10-0-137-19 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 17:11:55.052275 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.052188    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 17:11:55.060296 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060259    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:11:55.060296 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060290    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:11:55.060296 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060298    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:11:55.060296 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060301    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060305    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060310    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060313    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060317    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060320    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060323    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060326    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060329    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060332    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060335    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060339    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060345    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060349    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060354    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060358    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060361    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060364    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060368    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:11:55.060459 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060370    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060373    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060376    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060379    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060386    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060389    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060392    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060396    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060398    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060401    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060404    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060407    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060410    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060413    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060416    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060421    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060427    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060430    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060434    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:11:55.060923 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060437    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060440    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060443    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060446    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060449    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060452    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060455    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060458    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060461    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060464    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060469    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060472    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060475    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060478    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060482    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060485    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060487    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060490    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060493    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060496    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:11:55.061394 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060499    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060504    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060507    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060509    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060513    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060515    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060518    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060522    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060525    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060527    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060530    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060533    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060535    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060538    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060543    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060546    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060549    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060552    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060554    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060557    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:11:55.061930 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060560    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060563    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060565    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060568    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060571    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060573    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060576    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060581    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060583    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060586    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060589    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060591    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060594    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060597    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060599    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060602    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060605    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.060608    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061879    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:11:55.062422 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061889    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061894    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061897    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061900    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061903    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061906    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061910    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061912    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061916    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061918    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061921    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061924    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061927    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061930    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061932    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061935    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061938    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061941    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061944    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061946    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:11:55.062903 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061949    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061952    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061954    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061957    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061959    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061962    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061964    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061967    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061970    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061972    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061975    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061978    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061982    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061985    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061988    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061990    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061993    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061996    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.061998    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062002    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062004    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:11:55.063397 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062007    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062009    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062012    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062015    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062018    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062020    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062022    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062025    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062027    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062030    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062032    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062035    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062037    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062040    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062042    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062045    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062048    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062051    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062053    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062056    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:11:55.063944 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062058    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062062    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062066    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062070    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062073    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062076    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062079    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062081    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062084    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062088    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062091    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062095    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062097    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062100    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062102    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062105    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062107    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062112    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:11:55.064423 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062116    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062119    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062122    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062126    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062129    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062132    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062134    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062137    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062139    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062142    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062145    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062147    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062149    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062152    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062154    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062157    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062160    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062163    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062166    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062544    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:11:55.064900 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062549    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062552    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062555    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062557    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062560    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062563    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062566    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062568    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062571    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062573    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062576    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062578    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062581    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062584    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062586    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062589    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062592    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062594    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062597    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:11:55.065405 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062599    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062603    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062605    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062608    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062610    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062613    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062615    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062617    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062620    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062623    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062626    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062629    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062631    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062634    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062636    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062639    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062642    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062644    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062647    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062649    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:11:55.065894 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062652    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062654    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062657    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062674    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062677    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062680    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062684    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062687    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062690    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062694    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062697    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062700    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062703    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062705    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062708    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062711    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062713    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062717    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062721    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:11:55.066391 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062723    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062726    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062729    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062732    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062735    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062738    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062740    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062743    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062746    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062748    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062751    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062753    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062757    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062760    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062764    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062767    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062770    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062772    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062775    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062779    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:11:55.066908 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062782    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062785    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062787    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062790    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062793    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062795    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062798    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062801    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062804    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062806    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062808    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062811    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062814    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062816    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062819    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062821    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062824    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062826    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062828    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.062831    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:11:55.067402 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062899    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062930    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062935    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062945    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062949    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062952    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062956    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062960    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062963    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062966    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062970    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062973    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062976    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062979    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062984    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062987    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062991    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062994    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.062999    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063002    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063005    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063008    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063011    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063018    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 17:11:55.067910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063021    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063024    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063028    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063031    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063034    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063037    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063040    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063043    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063046    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063049    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063052    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063055    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063058    2556 flags.go:64] FLAG: --enable-server="true"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063060    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063064    2556 flags.go:64] FLAG: --event-burst="100"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063067    2556 flags.go:64] FLAG: --event-qps="50"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063070    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063073    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063076    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063079    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063082    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063085    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063091    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063094    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063097    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 17:11:55.068502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063102    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063105    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063108    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063110    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063113    2556 flags.go:64] FLAG: --feature-gates=""
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063116    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063120    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063123    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063126    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063129    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063132    2556 flags.go:64] FLAG: --help="false"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063135    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063138    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063141    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063143    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063147    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063150    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063153    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063155    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063158    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063161    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063164    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063167    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063170    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 10 17:11:55.069150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063172    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063175    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063178    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063180    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063183    2556 flags.go:64] FLAG: --lock-file=""
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063187    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063190    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063193    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063199    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063202    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063205    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063208    2556 flags.go:64] FLAG: --logging-format="text"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063211    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063214    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063216    2556 flags.go:64] FLAG: --manifest-url=""
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063219    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063223    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063226    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063230    2556 flags.go:64] FLAG: --max-pods="110"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063233    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063236    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063239    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063242    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063245    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063248    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 17:11:55.069750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063251    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063258    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063261    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063263    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063266    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063269    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063275    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063278    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063281    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063284    2556 flags.go:64] FLAG: --port="10250"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063287    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063290    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d429d49882b8c684"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063292    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063296    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063299    2556 flags.go:64] FLAG: --register-node="true"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063304    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063307    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063310    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063313    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063316    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063320    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063323    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063326    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063329    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063332    2556 flags.go:64] FLAG: --runonce="false"
Sep 10 17:11:55.070401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063335    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063338    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063341    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063344    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063347    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063350    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063353    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063355    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063358    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063361    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063364    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063367    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063370    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063372    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063375    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063380    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063383    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063386    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063390    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063393    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063397    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063400    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063405    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063408    2556 flags.go:64] FLAG: --v="2"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063416    2556 flags.go:64] FLAG: --version="false"
Sep 10 17:11:55.071160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063420    2556 flags.go:64] FLAG: --vmodule=""
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063424    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.063427    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063539    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063543    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063546    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063549    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063551    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063554    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063556    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063559    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063561    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063564    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063567    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063570    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063572    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063575    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063577    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063580    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:11:55.071807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063582    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063585    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063588    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063590    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063594    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063598    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063601    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063604    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063608    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063611    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063615    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063617    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063620    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063623    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063628    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063631    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063634    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063636    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063639    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:11:55.072278 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063643    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063646    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063649    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063652    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063654    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063657    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063674    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063677    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063680    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063682    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063685    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063687    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063690    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063692    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063695    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063697    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063700    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063702    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063705    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063708    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:11:55.072768 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063711    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063714    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063717    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063721    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063724    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063727    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063731    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063734    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063737    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063740    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063743    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063746    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063748    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063751    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063753    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063756    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063759    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063761    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063763    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:11:55.073251 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063766    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063768    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063771    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063773    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063776    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063778    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063780    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063783    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063786    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063788    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063791    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063793    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063796    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063798    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063803    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063805    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063809    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063812    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063814    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063817    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:11:55.073733 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063819    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063822    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063825    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063827    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.063830    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.064635    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.071884    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.071899    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.071918    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.071923    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.071982    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.071986    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.071990    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.071993    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:11:55.074234 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.071996    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.071999    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072002    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072004    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072007    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072010    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072013    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072015    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072018    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072021    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072023    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072028    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072031    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072034    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072037    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072040    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072043    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072046    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072048    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:11:55.074577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072051    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072057    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072060    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072063    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072066    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072068    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072073    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072077    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072080    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072083    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072086    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072089    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072092    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072095    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072097    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072100    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072103    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072106    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072108    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072111    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:11:55.075056 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072113    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072116    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072119    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072121    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072124    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072127    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072131    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072134    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072137    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072140    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072142    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072145    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072147    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072150    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072155    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072159    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072162    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072164    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072167    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:11:55.075560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072170    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072172    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072175    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072178    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072180    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072183    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072186    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072188    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072190    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072193    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072195    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072198    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072201    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072203    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072206    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072208    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072211    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072214    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072216    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072219    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:11:55.076053 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072221    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072224    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072227    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072229    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072232    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072234    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072237    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072240    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072244    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072246    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072248    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072251    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072254    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072257    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072259    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072262    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:11:55.076542 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072264    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.072269    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072379    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072384    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072387    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072390    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072394    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072396    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072400    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072402    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072405    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072408    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072411    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072414    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072417    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 17:11:55.076951 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072419    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072422    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072425    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072427    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072430    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072432    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072435    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072437    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072440    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072443    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072446    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072449    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072451    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072454    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072457    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072459    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072462    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072464    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072467    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072470    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072472    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 17:11:55.077315 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072475    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072478    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072480    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072482    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072485    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072487    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072490    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072493    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072495    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072498    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072500    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072503    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072505    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072508    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072510    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072513    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072515    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072518    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072520    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 17:11:55.077830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072523    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072525    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072528    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072531    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072535    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072539    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072542    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072545    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072547    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072550    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072552    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072555    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072557    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072560    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072563    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072565    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072567    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072570    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072573    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 17:11:55.078289 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072575    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072578    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072581    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072583    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072586    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072588    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072591    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072593    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072596    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072598    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072601    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072604    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072606    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072609    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072612    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072616    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072619    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072622    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072625    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 17:11:55.078896 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072627    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072630    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072632    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072635    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072638    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072640    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072643    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:55.072645    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.072650    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.072772    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.075125    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.076053    2556 server.go:1019] "Starting client certificate rotation"
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.076147    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.076185    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 17:11:55.079354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.078917    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:11:55.080900 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.080883    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 17:11:55.090606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.090585    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 10 17:11:55.093708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.093691    2556 log.go:25] "Validated CRI v1 image API"
Sep 10 17:11:55.094477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.094464    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 17:11:55.097357 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.097340    2556 fs.go:135] Filesystem UUIDs: map[72c4514d-7cc6-4421-8276-8c28fa7bedc8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ef06ac09-96c2-4f6b-b827-56d1d8877fb8:/dev/nvme0n1p4]
Sep 10 17:11:55.097445 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.097357    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 17:11:55.101628 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.101605    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:11:55.103446 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.103332    2556 manager.go:216] Machine: {Timestamp:2026-09-10 17:11:55.102113232 +0000 UTC m=+0.322237156 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103968 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e48602ea04f9be4965889445f3e1 SystemUUID:ec28e486-02ea-04f9-be49-65889445f3e1 BootID:71a12f57-ade3-4e01-9cfc-7e345c33ea27 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:36:6f:8a:0f:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:36:6f:8a:0f:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:e6:cd:43:cc:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 17:11:55.103446 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.103441    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 17:11:55.103570 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.103519    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 17:11:55.104608 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.104585    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 17:11:55.104786 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.104611    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 17:11:55.105494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.105481    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 17:11:55.105547 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.105499    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 17:11:55.105547 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.105509    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:11:55.106056 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.106044    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 17:11:55.107890 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.107878    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 17:11:55.111700 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111687    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:11:55.111841 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111830    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 17:11:55.111898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111867    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 17:11:55.111898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111891    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 17:11:55.111989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111908    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 17:11:55.111989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111921    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 10 17:11:55.111989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.111934    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 17:11:55.112281 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.112267    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:11:55.112340 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.112290    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 17:11:55.114464 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.114448    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 17:11:55.117083 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117070    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 17:11:55.117149 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117099    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 17:11:55.117830 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117819    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 17:11:55.117871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117837    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 17:11:55.117871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117843    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 17:11:55.117871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117849    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 17:11:55.117871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117857    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 17:11:55.117871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117862    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 17:11:55.117996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117939    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 17:11:55.117996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117945    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 17:11:55.117996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117959    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 17:11:55.117996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117965    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 17:11:55.117996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117973    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 17:11:55.117996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.117980    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 17:11:55.118701 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.118691    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 17:11:55.118701 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.118701    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 17:11:55.119845 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.119824    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 17:11:55.119845 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.119829    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 17:11:55.122603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.122581    2556 server.go:1268] "Started kubelet"
Sep 10 17:11:55.122694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.122642    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 17:11:55.122908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.122868    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 17:11:55.123284 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.122806    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 17:11:55.123363 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.123302    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 17:11:55.123702 ip-10-0-137-19 systemd[1]: Started Kubernetes Kubelet.
Sep 10 17:11:55.125634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.124713    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 17:11:55.130086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.128233    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fxz2q"
Sep 10 17:11:55.130086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.129722    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 17:11:55.130086 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.129920    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 17:11:55.133396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.133377    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fxz2q"
Sep 10 17:11:55.134214 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.134196    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 17:11:55.135586 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.135571    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 17:11:55.135726 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.135706    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.136459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136442    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 17:11:55.136459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136461    2556 factory.go:55] Registering systemd factory
Sep 10 17:11:55.136617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136471    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 10 17:11:55.136617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136513    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 17:11:55.136617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136544    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 17:11:55.136617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136563    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 17:11:55.136617 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.135203    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-19.ec2.internal.18d404e0b4a0e588  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-19.ec2.internal,UID:ip-10-0-137-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-19.ec2.internal,},FirstTimestamp:2026-09-10 17:11:55.122558344 +0000 UTC m=+0.342682267,LastTimestamp:2026-09-10 17:11:55.122558344 +0000 UTC m=+0.342682267,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-19.ec2.internal,}"
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136646    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136653    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136719    2556 factory.go:153] Registering CRI-O factory
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136735    2556 factory.go:223] Registration of the crio container factory successfully
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136774    2556 factory.go:103] Registering Raw factory
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.136792    2556 manager.go:1196] Started watching for new ooms in manager
Sep 10 17:11:55.137762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.137710    2556 manager.go:349] Starting recovery of all containers
Sep 10 17:11:55.138119 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.138010    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 17:11:55.139702 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.138135    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 17:11:55.149814 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.149648    2556 manager.go:354] Recovery completed
Sep 10 17:11:55.155307 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.155294    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.156987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.156969    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.157074 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.157000    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.157074 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.157012    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.157526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.157513    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.157526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.157518    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 17:11:55.157654 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.157530    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 17:11:55.157654 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.157547    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 17:11:55.158222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.158207    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.158298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.158236    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.158298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.158249    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.158887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.158874    2556 policy_none.go:49] "None policy: Start"
Sep 10 17:11:55.158940 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.158894    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 17:11:55.158940 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.158908    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 17:11:55.160266 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.160255    2556 policy_none.go:47] "Start"
Sep 10 17:11:55.211272 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211249    2556 manager.go:337] "Starting Device Plugin manager"
Sep 10 17:11:55.211412 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.211302    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 17:11:55.211412 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211316    2556 server.go:87] "Starting device plugin registration server"
Sep 10 17:11:55.211701 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211686    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 17:11:55.211751 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211704    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 17:11:55.211870 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211855    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 17:11:55.211947 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211936    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 17:11:55.211947 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.211946    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 17:11:55.212499 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.212479    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 17:11:55.212591 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.212530    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.263644 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.263605    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 17:11:55.264867 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.264847    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 17:11:55.264942 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.264895    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 17:11:55.264942 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.264917    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 17:11:55.265038 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.265009    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 17:11:55.266935 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.266914    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:11:55.312830 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.312754    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.313598 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.313583    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.313694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.313617    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.313694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.313634    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.313694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.313680    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.319355 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.319340    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.319416 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.319361    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-19.ec2.internal\": node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.332278 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.332262    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.365466 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.365428    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"]
Sep 10 17:11:55.365546 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.365504    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.367073 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.367059    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.367142 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.367087    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.367142 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.367103    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.368749 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.368736    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.368867 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.368854    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.368912 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.368883    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.369945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.369928    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.370032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.369959    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.370032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.369928    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.370032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.369995    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.370032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.370006    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.370032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.369971    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.371700 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.371685    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.371747 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.371718    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 17:11:55.372390 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.372375    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 17:11:55.372461 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.372401    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 17:11:55.372461 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.372413    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 17:11:55.393195 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.393174    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-19.ec2.internal\" not found" node="ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.397130 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.397115    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-19.ec2.internal\" not found" node="ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.432768 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.432750    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.438482 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.438466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.438535 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.438490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.438535 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.438507    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70b70169d86c3bcf2d0a1e114ed46d9a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-19.ec2.internal\" (UID: \"70b70169d86c3bcf2d0a1e114ed46d9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.533506 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.533483    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.538852 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.538833    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.538933 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.538874    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.538933 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.538889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70b70169d86c3bcf2d0a1e114ed46d9a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-19.ec2.internal\" (UID: \"70b70169d86c3bcf2d0a1e114ed46d9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.538933 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.538929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.539066 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.538976    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/99981a6272193d33eff6882647ea56fb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal\" (UID: \"99981a6272193d33eff6882647ea56fb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.539066 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.538978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/70b70169d86c3bcf2d0a1e114ed46d9a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-19.ec2.internal\" (UID: \"70b70169d86c3bcf2d0a1e114ed46d9a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.634212 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.634145    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.696673 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.696634    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.701345 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:55.701328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:55.735090 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.735061    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.837922 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.837886    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:55.939095 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:55.939016    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:56.039769 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:56.039737    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:56.075442 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.075411    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 17:11:56.075910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.075552    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 17:11:56.140860 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:56.140836    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:56.140860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.140848    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 17:06:56 +0000 UTC" deadline="2028-02-23 03:35:34.372990437 +0000 UTC"
Sep 10 17:11:56.141007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.140871    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12730h23m38.232122511s"
Sep 10 17:11:56.241855 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:56.241795    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-19.ec2.internal\" not found"
Sep 10 17:11:56.255732 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.255709    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:11:56.288042 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.288019    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:11:56.330850 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:56.330813    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b70169d86c3bcf2d0a1e114ed46d9a.slice/crio-51853b1f18bc984b6c60204f555746d29ce13e76686ba9ff723380fd7335123c WatchSource:0}: Error finding container 51853b1f18bc984b6c60204f555746d29ce13e76686ba9ff723380fd7335123c: Status 404 returned error can't find the container with id 51853b1f18bc984b6c60204f555746d29ce13e76686ba9ff723380fd7335123c
Sep 10 17:11:56.331160 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:56.331136    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99981a6272193d33eff6882647ea56fb.slice/crio-9e7bcc69945c3febfd8359c3d97c72dd91bcfb874b7be990065d73a67e44d9a1 WatchSource:0}: Error finding container 9e7bcc69945c3febfd8359c3d97c72dd91bcfb874b7be990065d73a67e44d9a1: Status 404 returned error can't find the container with id 9e7bcc69945c3febfd8359c3d97c72dd91bcfb874b7be990065d73a67e44d9a1
Sep 10 17:11:56.335918 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.335891    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:56.336768 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.336728    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:11:56.349563 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.349546    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:11:56.351729 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.351709    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal"
Sep 10 17:11:56.360971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.360956    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 17:11:56.435666 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:56.435647    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:11:57.113526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.113492    2556 apiserver.go:52] "Watching apiserver"
Sep 10 17:11:57.125110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.125079    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 17:11:57.126549 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.126516    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-9dl82","kube-system/konnectivity-agent-nnk4v","openshift-cluster-node-tuning-operator/tuned-tkbdl","openshift-image-registry/node-ca-2t5g6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal","openshift-multus/multus-additional-cni-plugins-2gcq6","openshift-multus/multus-n6brp","openshift-network-diagnostics/network-check-target-wxqdz","openshift-ovn-kubernetes/ovnkube-node-wxqt9","kube-system/global-pull-secret-syncer-6fwnl","kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j","openshift-dns/node-resolver-nxqp5","openshift-multus/network-metrics-daemon-pz5km"]
Sep 10 17:11:57.130358 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.130287    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.132998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.132947    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:11:57.133144 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.133039    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:11:57.133415 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.133392    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 17:11:57.133550 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.133518    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 17:11:57.134055 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.134028    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.134172 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.134062    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-g8q78\""
Sep 10 17:11:57.134172 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.134120    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.134342 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.134324    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 17:11:57.135550 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.135530    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.137782 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.137763    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.138024 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.138004    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.138176 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.137875    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5t7wm\""
Sep 10 17:11:57.138401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.138382    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.140582 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.140565    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.141320 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.141298    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.141491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.141476    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.141625 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.141577    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4g7cn\""
Sep 10 17:11:57.141714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.141696    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 17:11:57.143135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.143104    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qjhbt\""
Sep 10 17:11:57.143251 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.143234    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.143251 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.143246    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 17:11:57.143580 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.143520    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.145095 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.145072    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.148057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.147566    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 17:11:57.148057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.147569    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 17:11:57.148057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.147851    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x4tfw\""
Sep 10 17:11:57.148057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.147985    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.152129 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.152108    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 17:11:57.152560 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.152541    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 17:11:57.152807 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.152756    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 17:11:57.152895 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.152866    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6m6lz\""
Sep 10 17:11:57.152990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.152973    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 17:11:57.153151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153134    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.153262 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153241    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.153360 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153342    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.153633 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153613    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.153633 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153626    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1caed0f-747d-48e1-b90a-a07ac6b1177b-cni-binary-copy\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.153795 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153655    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-daemon-config\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.153795 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-kubernetes\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.153795 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153750    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-sys\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.153795 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-etc-kubernetes\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:11:57.154007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153838    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-run\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-cnibin\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153937    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-hostroot\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.153964    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-conf-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-modprobe-d\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-host\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154056    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4ad1c79-559d-4d22-b205-47c091cee22c-tmp\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be1264a-5add-475d-9fe4-adcf3db28e24-serviceca\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-kubelet\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysconfig\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154147    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-socket-dir-parent\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154180    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lmhl\" (UniqueName: \"kubernetes.io/projected/e1caed0f-747d-48e1-b90a-a07ac6b1177b-kube-api-access-9lmhl\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-systemd\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154261    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45phv\" (UniqueName: \"kubernetes.io/projected/9be1264a-5add-475d-9fe4-adcf3db28e24-kube-api-access-45phv\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154297    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-system-cni-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysctl-d\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154379    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-var-lib-kubelet\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be1264a-5add-475d-9fe4-adcf3db28e24-host\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154427    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-cni-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-k8s-cni-cncf-io\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154477    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-netns\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-cni-bin\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-cni-multus\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-multus-certs\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.154915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysctl-conf\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154635    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2f7\" (UniqueName: \"kubernetes.io/projected/c4ad1c79-559d-4d22-b205-47c091cee22c-kube-api-access-jk2f7\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-lib-modules\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.154915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.154679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-tuned\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.155696 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.155672    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 17:11:57.155806 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.155694    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.155936 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.155916    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7p7jd\""
Sep 10 17:11:57.156850 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.156568    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 17:11:57.156850 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.156611    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.156850 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.156680    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.156850 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.156775    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 17:11:57.156850 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.156679    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:11:57.157161 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.157014    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jkgw6\""
Sep 10 17:11:57.159161 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.159037    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.159161 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.159130    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:11:57.161446 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.161423    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.164047 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.164022    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 17:11:57.164047 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.164037    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 17:11:57.164259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.164242    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 17:11:57.164346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.164335    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rncgg\""
Sep 10 17:11:57.175060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.175025    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-49kdc"
Sep 10 17:11:57.182951 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.182928    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-49kdc"
Sep 10 17:11:57.228581 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.228544    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 17:11:57.237903 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.237878    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 17:11:57.255323 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-var-lib-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.255491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255334    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-cni-netd\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.255491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255367    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysconfig\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.255491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255393    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovn-node-metrics-cert\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.255491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255421    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45phv\" (UniqueName: \"kubernetes.io/projected/9be1264a-5add-475d-9fe4-adcf3db28e24-kube-api-access-45phv\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.255491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255446    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93248a9-6223-4627-a97a-e57b5f22539c-host-slash\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.255491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255469    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-systemd\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255492    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29c79ac8-60f9-434a-9687-28a323a15418-dbus\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46a93d23-62be-48a7-ad9c-ffa56649cbcd-konnectivity-ca\") pod \"konnectivity-agent-nnk4v\" (UID: \"46a93d23-62be-48a7-ad9c-ffa56649cbcd\") " pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255548    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-registration-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255575    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-system-cni-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysctl-d\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be1264a-5add-475d-9fe4-adcf3db28e24-host\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255648    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.255736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-cnibin\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.256155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255759    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.256155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255789    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-device-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.256155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-k8s-cni-cncf-io\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.255978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysconfig\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.256155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-system-cni-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256146    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9be1264a-5add-475d-9fe4-adcf3db28e24-host\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysctl-d\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-cni-bin\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-k8s-cni-cncf-io\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-multus-certs\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256273    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-cni-bin\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256312    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysctl-conf\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-multus-certs\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256357    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2f7\" (UniqueName: \"kubernetes.io/projected/c4ad1c79-559d-4d22-b205-47c091cee22c-kube-api-access-jk2f7\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256386    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxdmn\" (UniqueName: \"kubernetes.io/projected/d93248a9-6223-4627-a97a-e57b5f22539c-kube-api-access-pxdmn\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kmzh\" (UniqueName: \"kubernetes.io/projected/14bbd0d2-4568-41dc-ad8f-6129d10314f8-kube-api-access-4kmzh\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.256468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24cw9\" (UniqueName: \"kubernetes.io/projected/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-kube-api-access-24cw9\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256476    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-tuned\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93248a9-6223-4627-a97a-e57b5f22539c-iptables-alerter-script\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256578    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/faefffb8-eb30-491d-9b6f-a4b10526bc17-kube-api-access-jd4wf\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256606    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-system-cni-dir\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256634    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1caed0f-747d-48e1-b90a-a07ac6b1177b-cni-binary-copy\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256658    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-daemon-config\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256682    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-kubernetes\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-sys\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256751    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovnkube-config\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46a93d23-62be-48a7-ad9c-ffa56649cbcd-agent-certs\") pod \"konnectivity-agent-nnk4v\" (UID: \"46a93d23-62be-48a7-ad9c-ffa56649cbcd\") " pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-socket-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256934    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-etc-kubernetes\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256960    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-kubelet\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.256982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-cni-bin\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29c79ac8-60f9-434a-9687-28a323a15418-kubelet-config\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.257110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-cnibin\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257058    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-hostroot\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-conf-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-modprobe-d\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257175    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4ad1c79-559d-4d22-b205-47c091cee22c-tmp\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-env-overrides\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjpr\" (UniqueName: \"kubernetes.io/projected/dcecaa29-0662-4242-81b6-94bf162b78fa-kube-api-access-tzjpr\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257253    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-sys\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-kubelet\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-etc-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rj9h\" (UniqueName: \"kubernetes.io/projected/37b41989-8f5c-495f-91d6-faa4f75c2e97-kube-api-access-6rj9h\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257439    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257463    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-socket-dir-parent\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257489    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lmhl\" (UniqueName: \"kubernetes.io/projected/e1caed0f-747d-48e1-b90a-a07ac6b1177b-kube-api-access-9lmhl\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257517    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-systemd\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.257661 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257574    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovnkube-script-lib\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257603    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257629    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14bbd0d2-4568-41dc-ad8f-6129d10314f8-tmp-dir\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-var-lib-kubelet\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-run-netns\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-ovn\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-kubelet\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-etc-kubernetes\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-cnibin\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257863    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-systemd\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257890    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-node-log\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-sysctl-conf\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257921    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-kubernetes\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257939    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-hostroot\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-cni-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-conf-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-netns\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.258537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.257927    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-socket-dir-parent\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-modprobe-d\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258057    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-var-lib-kubelet\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258060    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-run-netns\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-cni-multus\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258111    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-cni-dir\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258216    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-lib-modules\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1caed0f-747d-48e1-b90a-a07ac6b1177b-host-var-lib-cni-multus\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258252    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-systemd-units\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258319    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14bbd0d2-4568-41dc-ad8f-6129d10314f8-hosts-file\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-log-socket\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-lib-modules\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258510    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-run\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-sys-fs\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.259476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258592    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-host\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258623    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be1264a-5add-475d-9fe4-adcf3db28e24-serviceca\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258654    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-slash\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.258944    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-run\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.259089    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1caed0f-747d-48e1-b90a-a07ac6b1177b-multus-daemon-config\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.259413    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1caed0f-747d-48e1-b90a-a07ac6b1177b-cni-binary-copy\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.259461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c4ad1c79-559d-4d22-b205-47c091cee22c-host\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.260381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.259503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9be1264a-5add-475d-9fe4-adcf3db28e24-serviceca\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.262848 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.262822    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c4ad1c79-559d-4d22-b205-47c091cee22c-etc-tuned\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.263167 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.263143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4ad1c79-559d-4d22-b205-47c091cee22c-tmp\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.267563 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.267541    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:11:57.267667 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.267568    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:11:57.267667 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.267583    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:11:57.267811 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.267678    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:11:57.76764475 +0000 UTC m=+2.987768684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:11:57.269472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.269244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45phv\" (UniqueName: \"kubernetes.io/projected/9be1264a-5add-475d-9fe4-adcf3db28e24-kube-api-access-45phv\") pod \"node-ca-2t5g6\" (UID: \"9be1264a-5add-475d-9fe4-adcf3db28e24\") " pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.271250 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.271218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lmhl\" (UniqueName: \"kubernetes.io/projected/e1caed0f-747d-48e1-b90a-a07ac6b1177b-kube-api-access-9lmhl\") pod \"multus-n6brp\" (UID: \"e1caed0f-747d-48e1-b90a-a07ac6b1177b\") " pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.272402 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.272379    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2f7\" (UniqueName: \"kubernetes.io/projected/c4ad1c79-559d-4d22-b205-47c091cee22c-kube-api-access-jk2f7\") pod \"tuned-tkbdl\" (UID: \"c4ad1c79-559d-4d22-b205-47c091cee22c\") " pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.273372 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.273272    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" event={"ID":"99981a6272193d33eff6882647ea56fb","Type":"ContainerStarted","Data":"9e7bcc69945c3febfd8359c3d97c72dd91bcfb874b7be990065d73a67e44d9a1"}
Sep 10 17:11:57.275136 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.275101    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" event={"ID":"70b70169d86c3bcf2d0a1e114ed46d9a","Type":"ContainerStarted","Data":"51853b1f18bc984b6c60204f555746d29ce13e76686ba9ff723380fd7335123c"}
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-sys-fs\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359102    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-slash\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-var-lib-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359154    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-cni-netd\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359181    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovn-node-metrics-cert\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93248a9-6223-4627-a97a-e57b5f22539c-host-slash\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359231    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-systemd\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359241    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-slash\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359255    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29c79ac8-60f9-434a-9687-28a323a15418-dbus\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359296    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46a93d23-62be-48a7-ad9c-ffa56649cbcd-konnectivity-ca\") pod \"konnectivity-agent-nnk4v\" (UID: \"46a93d23-62be-48a7-ad9c-ffa56649cbcd\") " pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-registration-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359390    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359412    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359421    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29c79ac8-60f9-434a-9687-28a323a15418-dbus\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-cnibin\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359462    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-cnibin\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.361536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-device-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359524    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-sys-fs\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359529    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxdmn\" (UniqueName: \"kubernetes.io/projected/d93248a9-6223-4627-a97a-e57b5f22539c-kube-api-access-pxdmn\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kmzh\" (UniqueName: \"kubernetes.io/projected/14bbd0d2-4568-41dc-ad8f-6129d10314f8-kube-api-access-4kmzh\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359587    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24cw9\" (UniqueName: \"kubernetes.io/projected/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-kube-api-access-24cw9\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359616    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93248a9-6223-4627-a97a-e57b5f22539c-iptables-alerter-script\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/faefffb8-eb30-491d-9b6f-a4b10526bc17-kube-api-access-jd4wf\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359670    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-system-cni-dir\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359698    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovnkube-config\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359722    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46a93d23-62be-48a7-ad9c-ffa56649cbcd-agent-certs\") pod \"konnectivity-agent-nnk4v\" (UID: \"46a93d23-62be-48a7-ad9c-ffa56649cbcd\") " pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-socket-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-kubelet\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359848    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-cni-bin\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29c79ac8-60f9-434a-9687-28a323a15418-kubelet-config\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359912    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-env-overrides\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.362490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359937    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjpr\" (UniqueName: \"kubernetes.io/projected/dcecaa29-0662-4242-81b6-94bf162b78fa-kube-api-access-tzjpr\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359962    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.359994    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-etc-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rj9h\" (UniqueName: \"kubernetes.io/projected/37b41989-8f5c-495f-91d6-faa4f75c2e97-kube-api-access-6rj9h\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovnkube-script-lib\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46a93d23-62be-48a7-ad9c-ffa56649cbcd-konnectivity-ca\") pod \"konnectivity-agent-nnk4v\" (UID: \"46a93d23-62be-48a7-ad9c-ffa56649cbcd\") " pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360139    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14bbd0d2-4568-41dc-ad8f-6129d10314f8-tmp-dir\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-run-netns\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-ovn\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360261    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-node-log\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360288    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360315    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-systemd-units\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360357    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14bbd0d2-4568-41dc-ad8f-6129d10314f8-hosts-file\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.363151 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-log-socket\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360564    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360675    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-log-socket\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-registration-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360772    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360808    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.360847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovnkube-config\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14bbd0d2-4568-41dc-ad8f-6129d10314f8-tmp-dir\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361145    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-var-lib-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-run-netns\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361173    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-ovn\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361194    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-cni-netd\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-node-log\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.361595    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.361663    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:11:57.363779 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.361695    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:57.861661858 +0000 UTC m=+3.081785771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361714    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-systemd-units\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.361745    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:11:57.861725047 +0000 UTC m=+3.081848969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362001    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-socket-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362057    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-kubelet\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-cni-bin\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362146    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29c79ac8-60f9-434a-9687-28a323a15418-kubelet-config\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362282    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-device-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362371    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37b41989-8f5c-495f-91d6-faa4f75c2e97-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/14bbd0d2-4568-41dc-ad8f-6129d10314f8-hosts-file\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362511    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-env-overrides\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362701    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d93248a9-6223-4627-a97a-e57b5f22539c-host-slash\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362800    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-etc-openvswitch\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.362915    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d93248a9-6223-4627-a97a-e57b5f22539c-iptables-alerter-script\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.364484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.363128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37b41989-8f5c-495f-91d6-faa4f75c2e97-system-cni-dir\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.365008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.363178    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-host-run-ovn-kubernetes\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.365008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.363291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.365008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.361676    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/faefffb8-eb30-491d-9b6f-a4b10526bc17-run-systemd\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.365008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.363922    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovnkube-script-lib\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.365008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.363972    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46a93d23-62be-48a7-ad9c-ffa56649cbcd-agent-certs\") pod \"konnectivity-agent-nnk4v\" (UID: \"46a93d23-62be-48a7-ad9c-ffa56649cbcd\") " pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.365465 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.365443    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/faefffb8-eb30-491d-9b6f-a4b10526bc17-ovn-node-metrics-cert\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.372572 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.372541    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rj9h\" (UniqueName: \"kubernetes.io/projected/37b41989-8f5c-495f-91d6-faa4f75c2e97-kube-api-access-6rj9h\") pod \"multus-additional-cni-plugins-2gcq6\" (UID: \"37b41989-8f5c-495f-91d6-faa4f75c2e97\") " pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.372862 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.372814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4wf\" (UniqueName: \"kubernetes.io/projected/faefffb8-eb30-491d-9b6f-a4b10526bc17-kube-api-access-jd4wf\") pod \"ovnkube-node-wxqt9\" (UID: \"faefffb8-eb30-491d-9b6f-a4b10526bc17\") " pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.373181 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.373155    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxdmn\" (UniqueName: \"kubernetes.io/projected/d93248a9-6223-4627-a97a-e57b5f22539c-kube-api-access-pxdmn\") pod \"iptables-alerter-9dl82\" (UID: \"d93248a9-6223-4627-a97a-e57b5f22539c\") " pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.374399 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.374095    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjpr\" (UniqueName: \"kubernetes.io/projected/dcecaa29-0662-4242-81b6-94bf162b78fa-kube-api-access-tzjpr\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.374399 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.374312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24cw9\" (UniqueName: \"kubernetes.io/projected/3a06cf44-f72c-4944-ae2d-f2605ff1f5c4-kube-api-access-24cw9\") pod \"aws-ebs-csi-driver-node-vs96j\" (UID: \"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.374399 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.374348    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kmzh\" (UniqueName: \"kubernetes.io/projected/14bbd0d2-4568-41dc-ad8f-6129d10314f8-kube-api-access-4kmzh\") pod \"node-resolver-nxqp5\" (UID: \"14bbd0d2-4568-41dc-ad8f-6129d10314f8\") " pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.444322 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.444277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n6brp"
Sep 10 17:11:57.453216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.453188    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tkbdl"
Sep 10 17:11:57.463779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.463754    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2t5g6"
Sep 10 17:11:57.471727 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.471705    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9dl82"
Sep 10 17:11:57.483519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.483494    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gcq6"
Sep 10 17:11:57.492450 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.492422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:11:57.500263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.500242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nxqp5"
Sep 10 17:11:57.509128 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.509098    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:11:57.516903 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.516876    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j"
Sep 10 17:11:57.865067 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.865035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:57.865259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.865084    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:57.865259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:57.865108    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:11:57.865259 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865217    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:11:57.865259 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865230    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:11:57.865259 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865253    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:11:57.865489 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865266    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:11:57.865489 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865283    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:11:58.865266845 +0000 UTC m=+4.085390761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:11:57.865489 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865321    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:11:58.865304592 +0000 UTC m=+4.085428527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:11:57.865489 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865217    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:11:57.865489 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:57.865388    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:11:58.865372134 +0000 UTC m=+4.085496070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:11:58.186804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:58.186715    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:06:58 +0000 UTC" deadline="2028-05-11 12:55:17.612414063 +0000 UTC"
Sep 10 17:11:58.186804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:58.186748    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14611h43m19.425670782s"
Sep 10 17:11:58.411505 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.411472    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a93d23_62be_48a7_ad9c_ffa56649cbcd.slice/crio-08f595644c07fd7efc461990d57ee4dd2d5abde34877aab0f2f93bf04aa634d0 WatchSource:0}: Error finding container 08f595644c07fd7efc461990d57ee4dd2d5abde34877aab0f2f93bf04aa634d0: Status 404 returned error can't find the container with id 08f595644c07fd7efc461990d57ee4dd2d5abde34877aab0f2f93bf04aa634d0
Sep 10 17:11:58.413346 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.413243    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd93248a9_6223_4627_a97a_e57b5f22539c.slice/crio-aef5a405303f752db3df3e6b76b706b7025e4db7828fa540bc217dba437fd782 WatchSource:0}: Error finding container aef5a405303f752db3df3e6b76b706b7025e4db7828fa540bc217dba437fd782: Status 404 returned error can't find the container with id aef5a405303f752db3df3e6b76b706b7025e4db7828fa540bc217dba437fd782
Sep 10 17:11:58.417130 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.417089    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1caed0f_747d_48e1_b90a_a07ac6b1177b.slice/crio-35c910f21d7567645716d4db65a8f5c33ec786084af2dacae4f579b08487d79c WatchSource:0}: Error finding container 35c910f21d7567645716d4db65a8f5c33ec786084af2dacae4f579b08487d79c: Status 404 returned error can't find the container with id 35c910f21d7567645716d4db65a8f5c33ec786084af2dacae4f579b08487d79c
Sep 10 17:11:58.418496 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.418469    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b41989_8f5c_495f_91d6_faa4f75c2e97.slice/crio-4886f91eb086009364600974eb28a714698c7680293a78bd911e1136c4cddbcc WatchSource:0}: Error finding container 4886f91eb086009364600974eb28a714698c7680293a78bd911e1136c4cddbcc: Status 404 returned error can't find the container with id 4886f91eb086009364600974eb28a714698c7680293a78bd911e1136c4cddbcc
Sep 10 17:11:58.418830 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.418810    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ad1c79_559d_4d22_b205_47c091cee22c.slice/crio-d40d1be2518700cf72c86f1d81a53d073a6714afd38d78b6de5c6d2965570ab2 WatchSource:0}: Error finding container d40d1be2518700cf72c86f1d81a53d073a6714afd38d78b6de5c6d2965570ab2: Status 404 returned error can't find the container with id d40d1be2518700cf72c86f1d81a53d073a6714afd38d78b6de5c6d2965570ab2
Sep 10 17:11:58.420877 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.420511    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be1264a_5add_475d_9fe4_adcf3db28e24.slice/crio-f31b15f566e40255cca5b2eae78a292d5fd8b9a06260c1fae72bfc81c5e2baad WatchSource:0}: Error finding container f31b15f566e40255cca5b2eae78a292d5fd8b9a06260c1fae72bfc81c5e2baad: Status 404 returned error can't find the container with id f31b15f566e40255cca5b2eae78a292d5fd8b9a06260c1fae72bfc81c5e2baad
Sep 10 17:11:58.421560 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.421522    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a06cf44_f72c_4944_ae2d_f2605ff1f5c4.slice/crio-4953c96e119f504890c35d8b6b9cabbc45b06ed362d20ec716947ff11690a682 WatchSource:0}: Error finding container 4953c96e119f504890c35d8b6b9cabbc45b06ed362d20ec716947ff11690a682: Status 404 returned error can't find the container with id 4953c96e119f504890c35d8b6b9cabbc45b06ed362d20ec716947ff11690a682
Sep 10 17:11:58.423075 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.423043    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaefffb8_eb30_491d_9b6f_a4b10526bc17.slice/crio-db2311daf3207ea45ad481546d169c33fef13b2cb880af4e109f84e52a5fc9c9 WatchSource:0}: Error finding container db2311daf3207ea45ad481546d169c33fef13b2cb880af4e109f84e52a5fc9c9: Status 404 returned error can't find the container with id db2311daf3207ea45ad481546d169c33fef13b2cb880af4e109f84e52a5fc9c9
Sep 10 17:11:58.425331 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:11:58.425309    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14bbd0d2_4568_41dc_ad8f_6129d10314f8.slice/crio-d39e0359e8d638872d5800cd8480e268b0311d05b46af7feac4400d4bdbb4131 WatchSource:0}: Error finding container d39e0359e8d638872d5800cd8480e268b0311d05b46af7feac4400d4bdbb4131: Status 404 returned error can't find the container with id d39e0359e8d638872d5800cd8480e268b0311d05b46af7feac4400d4bdbb4131
Sep 10 17:11:58.873137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:58.873096    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:58.873348 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:58.873176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:58.873348 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:58.873218    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:11:58.873348 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873275    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:11:58.873348 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873323    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:11:58.873348 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873342    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:11:58.873537 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873358    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:11:58.873537 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873366    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:12:00.873342126 +0000 UTC m=+6.093466064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:11:58.873537 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873372    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:11:58.873537 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873393    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:00.873379633 +0000 UTC m=+6.093503553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:11:58.873537 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:58.873418    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:12:00.873401301 +0000 UTC m=+6.093525229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:11:59.187644 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.187508    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 17:06:58 +0000 UTC" deadline="2028-06-09 16:11:30.508711574 +0000 UTC"
Sep 10 17:11:59.187644 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.187552    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15310h59m31.321164572s"
Sep 10 17:11:59.269776 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.268363    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:11:59.269776 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:59.268524    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:11:59.269776 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.269462    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:11:59.269776 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:59.269574    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:11:59.269776 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.269634    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:11:59.269776 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:11:59.269705    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:11:59.284157 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.283998    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j" event={"ID":"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4","Type":"ContainerStarted","Data":"4953c96e119f504890c35d8b6b9cabbc45b06ed362d20ec716947ff11690a682"}
Sep 10 17:11:59.293476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.293397    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2t5g6" event={"ID":"9be1264a-5add-475d-9fe4-adcf3db28e24","Type":"ContainerStarted","Data":"f31b15f566e40255cca5b2eae78a292d5fd8b9a06260c1fae72bfc81c5e2baad"}
Sep 10 17:11:59.302021 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.301977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nnk4v" event={"ID":"46a93d23-62be-48a7-ad9c-ffa56649cbcd","Type":"ContainerStarted","Data":"08f595644c07fd7efc461990d57ee4dd2d5abde34877aab0f2f93bf04aa634d0"}
Sep 10 17:11:59.320477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.320418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" event={"ID":"70b70169d86c3bcf2d0a1e114ed46d9a","Type":"ContainerStarted","Data":"ce4a18903ae47bf2a31ef507d9b13a6f27081047049a4aed5b55e3990c3e76c2"}
Sep 10 17:11:59.326551 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.326440    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"db2311daf3207ea45ad481546d169c33fef13b2cb880af4e109f84e52a5fc9c9"}
Sep 10 17:11:59.328617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.328588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tkbdl" event={"ID":"c4ad1c79-559d-4d22-b205-47c091cee22c","Type":"ContainerStarted","Data":"d40d1be2518700cf72c86f1d81a53d073a6714afd38d78b6de5c6d2965570ab2"}
Sep 10 17:11:59.336278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.335168    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-19.ec2.internal" podStartSLOduration=2.335150524 podStartE2EDuration="2.335150524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:11:59.335107796 +0000 UTC m=+4.555231737" watchObservedRunningTime="2026-09-10 17:11:59.335150524 +0000 UTC m=+4.555274462"
Sep 10 17:11:59.336278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.335250    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerStarted","Data":"4886f91eb086009364600974eb28a714698c7680293a78bd911e1136c4cddbcc"}
Sep 10 17:11:59.342818 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.342733    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n6brp" event={"ID":"e1caed0f-747d-48e1-b90a-a07ac6b1177b","Type":"ContainerStarted","Data":"35c910f21d7567645716d4db65a8f5c33ec786084af2dacae4f579b08487d79c"}
Sep 10 17:11:59.346964 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.346894    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9dl82" event={"ID":"d93248a9-6223-4627-a97a-e57b5f22539c","Type":"ContainerStarted","Data":"aef5a405303f752db3df3e6b76b706b7025e4db7828fa540bc217dba437fd782"}
Sep 10 17:11:59.355415 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.355383    2556 generic.go:356] "Generic (PLEG): container finished" podID="99981a6272193d33eff6882647ea56fb" containerID="9200f7131aa0c554c58f27c20a07faeb02d2102cd09dc1b183f162ff0bedcc77" exitCode=0
Sep 10 17:11:59.355540 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.355467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" event={"ID":"99981a6272193d33eff6882647ea56fb","Type":"ContainerDied","Data":"9200f7131aa0c554c58f27c20a07faeb02d2102cd09dc1b183f162ff0bedcc77"}
Sep 10 17:11:59.370687 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:11:59.370517    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxqp5" event={"ID":"14bbd0d2-4568-41dc-ad8f-6129d10314f8","Type":"ContainerStarted","Data":"d39e0359e8d638872d5800cd8480e268b0311d05b46af7feac4400d4bdbb4131"}
Sep 10 17:12:00.388929 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:00.388800    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" event={"ID":"99981a6272193d33eff6882647ea56fb","Type":"ContainerStarted","Data":"009666b6cd1253bb62b3bb105c1072a383d828be9922777c5f7904c62c522292"}
Sep 10 17:12:00.891513 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:00.891476    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:00.891694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:00.891555    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:00.891694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:00.891605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:00.891844 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.891752    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:00.891844 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.891818    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:04.891799698 +0000 UTC m=+10.111923614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:00.892275 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.892252    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:12:00.892361 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.892284    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:12:00.892361 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.892298    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:00.892361 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.892351    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:12:04.892334026 +0000 UTC m=+10.112457942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:00.892549 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.892422    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:00.892549 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:00.892458    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:12:04.892445468 +0000 UTC m=+10.112569386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:01.266636 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:01.265850    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:01.266636 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:01.266028    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:01.266636 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:01.265850    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:01.266636 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:01.266504    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:01.267616 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:01.267209    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:01.267616 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:01.267325    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:03.266656 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:03.266616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:03.267152 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:03.266775    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:03.267236 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:03.267214    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:03.267558 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:03.267331    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:03.267558 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:03.267414    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:03.267558 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:03.267512    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:04.929490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:04.929433    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:04.929490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:04.929494    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:04.929553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929722    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929793    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:12:12.929772157 +0000 UTC m=+18.149896084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929806    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929875    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:12.929856058 +0000 UTC m=+18.149979986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929966    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929985    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.929999    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:04.930482 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:04.930035    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:12:12.930023326 +0000 UTC m=+18.150147265 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:05.267715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:05.266969    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:05.267715 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:05.267084    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:05.267715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:05.267467    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:05.267715 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:05.267563    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:05.268042 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:05.267798    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:05.268042 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:05.267910    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:07.266770 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:07.265782    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:07.266770 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:07.265917    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:07.266770 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:07.265789    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:07.266770 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:07.265970    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:07.266770 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:07.266042    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:07.266770 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:07.266115    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:09.265499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:09.265461    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:09.265971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:09.265460    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:09.265971 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:09.265597    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:09.265971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:09.265473    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:09.265971 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:09.265682    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:09.265971 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:09.265760    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:11.265261 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:11.265226    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:11.265261 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:11.265256    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:11.265829 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:11.265271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:11.265829 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:11.265382    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:11.265829 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:11.265495    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:11.265829 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:11.265571    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:12.994853    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:12.994918    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:12.994951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.994975    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995037    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:12:28.995019476 +0000 UTC m=+34.215143402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995049    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995065    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995076    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995089    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995116    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:12:28.995103729 +0000 UTC m=+34.215227660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:12.995184 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:12.995138    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:12:28.995126826 +0000 UTC m=+34.215250736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:13.265199 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:13.265118    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:13.265361 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:13.265247    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:13.265361 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:13.265262    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:13.265361 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:13.265344    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:13.265514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:13.265380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:13.265514 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:13.265427    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:15.267035 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:15.266998    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:15.267477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:15.267087    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:15.267477 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:15.267111    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:15.267477 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:15.267190    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:15.267477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:15.267198    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:15.267477 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:15.267326    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:17.265371 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:17.265347    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:17.265724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:17.265352    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:17.265724 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:17.265454    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:17.265724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:17.265352    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:17.265724 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:17.265510    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:17.265724 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:17.265612    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:18.429989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.429571    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"a5f80add41a28d4cc280c27b1a87536342a7e35437d838abc7620be9d3766e5b"}
Sep 10 17:12:18.429989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.429954    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"6bfcf328b4a3c6a21c50566a118167fae06efd25f0f186ea73fb3398636eca31"}
Sep 10 17:12:18.430915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.430015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"10e9104e9706383ce5748b65e6c9f351cd023898badeb36e6eca7a01b56ab183"}
Sep 10 17:12:18.430915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.430064    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"9f846617187d25ed26993758dbfc766c4ff4f28795926fc183c31921fd45c43d"}
Sep 10 17:12:18.430915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.430112    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"48cd8d1cbcf4d46119b0b358eddf02ddc1d19b7dc19cd8ad6aac81c127b854eb"}
Sep 10 17:12:18.430915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.430157    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"f7ecdec8bd9126329dd12e94d783eadb86c51299a74258930ef4eb19650b42a0"}
Sep 10 17:12:18.431345 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.431316    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tkbdl" event={"ID":"c4ad1c79-559d-4d22-b205-47c091cee22c","Type":"ContainerStarted","Data":"063d430dc60cb785981a72c001c920895ee364a18445547ff2fce38ae7cc43ab"}
Sep 10 17:12:18.433050 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.433025    2556 generic.go:356] "Generic (PLEG): container finished" podID="37b41989-8f5c-495f-91d6-faa4f75c2e97" containerID="c726446612942d077d3031d5ef45b335784549f59da41f5a159fc25b97afeb50" exitCode=0
Sep 10 17:12:18.433162 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.433108    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerDied","Data":"c726446612942d077d3031d5ef45b335784549f59da41f5a159fc25b97afeb50"}
Sep 10 17:12:18.435795 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.435768    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n6brp" event={"ID":"e1caed0f-747d-48e1-b90a-a07ac6b1177b","Type":"ContainerStarted","Data":"069f26033da4b3f699b99ab3cf75e2acd2b5e7aa078eb4e33a40e5f1d9850dc3"}
Sep 10 17:12:18.437445 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.437420    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nxqp5" event={"ID":"14bbd0d2-4568-41dc-ad8f-6129d10314f8","Type":"ContainerStarted","Data":"b69e94ac512790f100997a7e05879f21973e2307b8458356ccd1deb9baaf0757"}
Sep 10 17:12:18.439203 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.439174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j" event={"ID":"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4","Type":"ContainerStarted","Data":"60b4088f7c66af74960e16ae8e81262af9c880a29ce1243107a2b7f52b649196"}
Sep 10 17:12:18.440565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.440541    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2t5g6" event={"ID":"9be1264a-5add-475d-9fe4-adcf3db28e24","Type":"ContainerStarted","Data":"746bbaa0e8d1a4ba54ff66a02f5baeabb216383b3914f5fcc2ef7f3d0adebbba"}
Sep 10 17:12:18.442040 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.442019    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nnk4v" event={"ID":"46a93d23-62be-48a7-ad9c-ffa56649cbcd","Type":"ContainerStarted","Data":"021189b1a3cc6d841ba292d2aeae951985f58d3585efb7a72d145573e3df33d9"}
Sep 10 17:12:18.450714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.450651    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tkbdl" podStartSLOduration=3.591083855 podStartE2EDuration="22.450634578s" totalImagesPullingTime="18.859550723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:18.449643236 +0000 UTC m=+23.669767168" watchObservedRunningTime="2026-09-10 17:12:18.450634578 +0000 UTC m=+23.670758509"
Sep 10 17:12:18.450871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.450845    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-19.ec2.internal" podStartSLOduration=21.45083457 podStartE2EDuration="21.45083457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:00.406104181 +0000 UTC m=+5.626228118" watchObservedRunningTime="2026-09-10 17:12:18.45083457 +0000 UTC m=+23.670958520"
Sep 10 17:12:18.465940 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.465902    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nxqp5" podStartSLOduration=3.483526373 podStartE2EDuration="22.465890494s" totalImagesPullingTime="18.982364121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:18.465556722 +0000 UTC m=+23.685680655" watchObservedRunningTime="2026-09-10 17:12:18.465890494 +0000 UTC m=+23.686014425"
Sep 10 17:12:18.481517 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.481473    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nnk4v" podStartSLOduration=3.655866209 podStartE2EDuration="22.48146064s" totalImagesPullingTime="18.825594431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:18.481089085 +0000 UTC m=+23.701213019" watchObservedRunningTime="2026-09-10 17:12:18.48146064 +0000 UTC m=+23.701584570"
Sep 10 17:12:18.504231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.504181    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n6brp" podStartSLOduration=3.281581305 podStartE2EDuration="22.504169887s" totalImagesPullingTime="19.222588582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:18.503858176 +0000 UTC m=+23.723982109" watchObservedRunningTime="2026-09-10 17:12:18.504169887 +0000 UTC m=+23.724293872"
Sep 10 17:12:18.517942 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.517899    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2t5g6" podStartSLOduration=3.53196099 podStartE2EDuration="22.517884195s" totalImagesPullingTime="18.985923205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:18.517567346 +0000 UTC m=+23.737691278" watchObservedRunningTime="2026-09-10 17:12:18.517884195 +0000 UTC m=+23.738008127"
Sep 10 17:12:18.896211 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:18.896030    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 17:12:19.222928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.222643    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T17:12:18.896205787Z","UUID":"b7442e83-e57c-4bbe-92aa-3b2167bcaee1","Handler":null,"Name":"","Endpoint":""}
Sep 10 17:12:19.224613 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.224587    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 17:12:19.224745 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.224624    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 17:12:19.265949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.265916    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:19.266127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.265935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:19.266127 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:19.266050    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:19.266127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.265935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:19.266281 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:19.266186    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:19.266332 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:19.266274    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:19.446745 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.446706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j" event={"ID":"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4","Type":"ContainerStarted","Data":"06c09a0e1ce018530acc98208f821166af2caa1cdc3e029b0080a68027c43b8d"}
Sep 10 17:12:19.448553 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.448521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9dl82" event={"ID":"d93248a9-6223-4627-a97a-e57b5f22539c","Type":"ContainerStarted","Data":"8fc096a8aaac664713113c1d8044e822ab7cccf8523eee7d6006dbef176cef96"}
Sep 10 17:12:19.462352 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:19.462304    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9dl82" podStartSLOduration=4.469561888 podStartE2EDuration="23.462290434s" totalImagesPullingTime="18.992728546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:19.461818957 +0000 UTC m=+24.681942890" watchObservedRunningTime="2026-09-10 17:12:19.462290434 +0000 UTC m=+24.682414366"
Sep 10 17:12:20.455567 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:20.455474    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"aeb519b6edf9daf3d46c2922cb966fa3429d69a46083801d5f34fe7445fdf6f0"}
Sep 10 17:12:20.457632 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:20.457592    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j" event={"ID":"3a06cf44-f72c-4944-ae2d-f2605ff1f5c4","Type":"ContainerStarted","Data":"78ff37a3197016830c3fdb0724415238d9f3d99ce4f1f16a75305cc0214a5f27"}
Sep 10 17:12:20.477699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:20.477631    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vs96j" podStartSLOduration=3.428943842 podStartE2EDuration="24.47761647s" totalImagesPullingTime="21.048672628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:20.476957882 +0000 UTC m=+25.697081818" watchObservedRunningTime="2026-09-10 17:12:20.47761647 +0000 UTC m=+25.697740426"
Sep 10 17:12:21.028038 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.028010    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:12:21.028505 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.028486    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:12:21.266001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.265969    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:21.266185 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.265978    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:21.266185 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:21.266098    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:21.266185 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.266121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:21.266344 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:21.266245    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:21.266344 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:21.266318    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:21.460114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.460025    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:12:21.460507 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:21.460404    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nnk4v"
Sep 10 17:12:23.265611 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.265402    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:23.266197 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.265404    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:23.266197 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:23.265646    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:23.266197 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:23.265726    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:23.266197 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.265422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:23.266197 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:23.265817    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:23.469837 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.469798    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" event={"ID":"faefffb8-eb30-491d-9b6f-a4b10526bc17","Type":"ContainerStarted","Data":"1ab756786b05fa9862288e85f4acaa84bda764c3561d823df103d1a04da62598"}
Sep 10 17:12:23.470065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.470038    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:12:23.471713 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.471692    2556 generic.go:356] "Generic (PLEG): container finished" podID="37b41989-8f5c-495f-91d6-faa4f75c2e97" containerID="1158a189f9344ac33128dbe614f94c0bee49aedd85f84fa18ec481f128903bd4" exitCode=0
Sep 10 17:12:23.471831 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.471776    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerDied","Data":"1158a189f9344ac33128dbe614f94c0bee49aedd85f84fa18ec481f128903bd4"}
Sep 10 17:12:23.485374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.485356    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:12:23.499415 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:23.499379    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9" podStartSLOduration=8.514207001 podStartE2EDuration="27.499368133s" totalImagesPullingTime="18.985161132s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:23.498262518 +0000 UTC m=+28.718386451" watchObservedRunningTime="2026-09-10 17:12:23.499368133 +0000 UTC m=+28.719492066"
Sep 10 17:12:24.476853 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.476818    2556 generic.go:356] "Generic (PLEG): container finished" podID="37b41989-8f5c-495f-91d6-faa4f75c2e97" containerID="5572a96273854662d56970ab92acdddadcbf6be69c26909ac37fd29b5524a78f" exitCode=0
Sep 10 17:12:24.477307 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.476910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerDied","Data":"5572a96273854662d56970ab92acdddadcbf6be69c26909ac37fd29b5524a78f"}
Sep 10 17:12:24.477607 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.477587    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:12:24.477717 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.477643    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:12:24.493690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.493653    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:12:24.788344 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.788270    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wxqdz"]
Sep 10 17:12:24.788530 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.788474    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:24.788606 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:24.788586    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:24.789066 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.789041    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6fwnl"]
Sep 10 17:12:24.789159 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.789148    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:24.789241 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:24.789216    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:24.790553 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.790528    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pz5km"]
Sep 10 17:12:24.790691 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:24.790639    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:24.790840 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:24.790814    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:25.481469 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:25.481386    2556 generic.go:356] "Generic (PLEG): container finished" podID="37b41989-8f5c-495f-91d6-faa4f75c2e97" containerID="fe46c4dd4098e45ce25266883d8d31c09c28472bbd8a7deec9745d01cb6f7e47" exitCode=0
Sep 10 17:12:25.481825 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:25.481466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerDied","Data":"fe46c4dd4098e45ce25266883d8d31c09c28472bbd8a7deec9745d01cb6f7e47"}
Sep 10 17:12:26.266044 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:26.266009    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:26.266223 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:26.266009    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:26.266223 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:26.266163    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:26.266223 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:26.266009    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:26.266398 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:26.266232    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:26.266398 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:26.266286    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:28.265551 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:28.265370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:28.265986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:28.265370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:28.265986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:28.265370    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:28.265986 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:28.265654    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:28.265986 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:28.265714    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:28.265986 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:28.265778    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:29.007550 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:29.007502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:29.007764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:29.007566    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:29.007764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:29.007614    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:29.007764 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007692    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:29.007764 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007739    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:29.007951 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007772    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 17:12:29.007951 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007794    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 17:12:29.007951 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007809    2556 projected.go:196] Error preparing data for projected volume kube-api-access-sv87q for pod openshift-network-diagnostics/network-check-target-wxqdz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:29.007951 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007776    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret podName:29c79ac8-60f9-434a-9687-28a323a15418 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:01.007753699 +0000 UTC m=+66.227877610 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret") pod "global-pull-secret-syncer-6fwnl" (UID: "29c79ac8-60f9-434a-9687-28a323a15418") : object "kube-system"/"original-pull-secret" not registered
Sep 10 17:12:29.007951 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007871    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs podName:dcecaa29-0662-4242-81b6-94bf162b78fa nodeName:}" failed. No retries permitted until 2026-09-10 17:13:01.007854304 +0000 UTC m=+66.227978216 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs") pod "network-metrics-daemon-pz5km" (UID: "dcecaa29-0662-4242-81b6-94bf162b78fa") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 17:12:29.007951 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:29.007887    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q podName:14070417-5047-4893-9c28-c96bbccec68b nodeName:}" failed. No retries permitted until 2026-09-10 17:13:01.007878021 +0000 UTC m=+66.228001934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sv87q" (UniqueName: "kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q") pod "network-check-target-wxqdz" (UID: "14070417-5047-4893-9c28-c96bbccec68b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 17:12:30.265297 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.265266    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:30.265818 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.265309    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:30.265818 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.265309    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:30.265818 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:30.265395    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wxqdz" podUID="14070417-5047-4893-9c28-c96bbccec68b"
Sep 10 17:12:30.265818 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:30.265513    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pz5km" podUID="dcecaa29-0662-4242-81b6-94bf162b78fa"
Sep 10 17:12:30.265818 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:30.265589    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6fwnl" podUID="29c79ac8-60f9-434a-9687-28a323a15418"
Sep 10 17:12:30.579142 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.579069    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-19.ec2.internal" event="NodeReady"
Sep 10 17:12:30.579301 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.579217    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 17:12:30.621887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.621856    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rh8lm"]
Sep 10 17:12:30.641037 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.640916    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7mbmg"]
Sep 10 17:12:30.641193 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.641111    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.643516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.643494    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 17:12:30.643964 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.643949    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-52r9p\""
Sep 10 17:12:30.644234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.644215    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 17:12:30.653946 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.653927    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rh8lm"]
Sep 10 17:12:30.654026 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.653959    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7mbmg"]
Sep 10 17:12:30.654063 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.654028    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.656555 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.656531    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 17:12:30.656658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.656601    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gj68b\""
Sep 10 17:12:30.656658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.656610    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 17:12:30.656658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.656614    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 17:12:30.720615 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.720584    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.720764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.720631    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2638b-880e-4a9a-b835-687ccc17feee-config-volume\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.720764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.720646    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2638b-880e-4a9a-b835-687ccc17feee-tmp-dir\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.720764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.720706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vgl\" (UniqueName: \"kubernetes.io/projected/10b2638b-880e-4a9a-b835-687ccc17feee-kube-api-access-b4vgl\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.821917 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.821884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.822079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.821927    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9svwf\" (UniqueName: \"kubernetes.io/projected/c7766669-96fc-4947-b998-827e01940bc8-kube-api-access-9svwf\") pod \"ingress-canary-7mbmg\" (UID: \"c7766669-96fc-4947-b998-827e01940bc8\") " pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.822079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.821958    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2638b-880e-4a9a-b835-687ccc17feee-config-volume\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.822079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.822005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2638b-880e-4a9a-b835-687ccc17feee-tmp-dir\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.822079 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:30.822029    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:12:30.822079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.822044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vgl\" (UniqueName: \"kubernetes.io/projected/10b2638b-880e-4a9a-b835-687ccc17feee-kube-api-access-b4vgl\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.822318 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:30.822094    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls podName:10b2638b-880e-4a9a-b835-687ccc17feee nodeName:}" failed. No retries permitted until 2026-09-10 17:12:31.322078653 +0000 UTC m=+36.542202564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls") pod "dns-default-rh8lm" (UID: "10b2638b-880e-4a9a-b835-687ccc17feee") : secret "dns-default-metrics-tls" not found
Sep 10 17:12:30.822318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.822125    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7766669-96fc-4947-b998-827e01940bc8-cert\") pod \"ingress-canary-7mbmg\" (UID: \"c7766669-96fc-4947-b998-827e01940bc8\") " pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.822427 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.822384    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10b2638b-880e-4a9a-b835-687ccc17feee-tmp-dir\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.822467 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.822423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10b2638b-880e-4a9a-b835-687ccc17feee-config-volume\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.852871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.852804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vgl\" (UniqueName: \"kubernetes.io/projected/10b2638b-880e-4a9a-b835-687ccc17feee-kube-api-access-b4vgl\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:30.894885 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.894852    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"]
Sep 10 17:12:30.920387 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.920359    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"]
Sep 10 17:12:30.920525 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.920477    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"
Sep 10 17:12:30.922736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.922718    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9svwf\" (UniqueName: \"kubernetes.io/projected/c7766669-96fc-4947-b998-827e01940bc8-kube-api-access-9svwf\") pod \"ingress-canary-7mbmg\" (UID: \"c7766669-96fc-4947-b998-827e01940bc8\") " pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.922824 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.922783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7766669-96fc-4947-b998-827e01940bc8-cert\") pod \"ingress-canary-7mbmg\" (UID: \"c7766669-96fc-4947-b998-827e01940bc8\") " pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.923008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.922993    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 10 17:12:30.923173 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.923144    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 10 17:12:30.923281 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.923160    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7npzj\""
Sep 10 17:12:30.925329 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.925310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c7766669-96fc-4947-b998-827e01940bc8-cert\") pod \"ingress-canary-7mbmg\" (UID: \"c7766669-96fc-4947-b998-827e01940bc8\") " pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.932074 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.932056    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9svwf\" (UniqueName: \"kubernetes.io/projected/c7766669-96fc-4947-b998-827e01940bc8-kube-api-access-9svwf\") pod \"ingress-canary-7mbmg\" (UID: \"c7766669-96fc-4947-b998-827e01940bc8\") " pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:30.961908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:30.961879    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7mbmg"
Sep 10 17:12:31.025026 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.024992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt9fg\" (UniqueName: \"kubernetes.io/projected/ba152442-477b-4c0f-959e-690826579bd1-kube-api-access-wt9fg\") pod \"migrator-fdbb8fdff-xzfp5\" (UID: \"ba152442-477b-4c0f-959e-690826579bd1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"
Sep 10 17:12:31.125785 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.125543    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt9fg\" (UniqueName: \"kubernetes.io/projected/ba152442-477b-4c0f-959e-690826579bd1-kube-api-access-wt9fg\") pod \"migrator-fdbb8fdff-xzfp5\" (UID: \"ba152442-477b-4c0f-959e-690826579bd1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"
Sep 10 17:12:31.133221 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.133197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt9fg\" (UniqueName: \"kubernetes.io/projected/ba152442-477b-4c0f-959e-690826579bd1-kube-api-access-wt9fg\") pod \"migrator-fdbb8fdff-xzfp5\" (UID: \"ba152442-477b-4c0f-959e-690826579bd1\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"
Sep 10 17:12:31.233423 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.233391    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"
Sep 10 17:12:31.284857 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.284822    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7mbmg"]
Sep 10 17:12:31.290450 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:31.290410    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7766669_96fc_4947_b998_827e01940bc8.slice/crio-732b9651b6b18fc299fe11536556581eff2db4f1433d4fdd26ac0c070f40bf80 WatchSource:0}: Error finding container 732b9651b6b18fc299fe11536556581eff2db4f1433d4fdd26ac0c070f40bf80: Status 404 returned error can't find the container with id 732b9651b6b18fc299fe11536556581eff2db4f1433d4fdd26ac0c070f40bf80
Sep 10 17:12:31.326915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.326886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:31.327121 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:31.327099    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:12:31.327229 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:31.327177    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls podName:10b2638b-880e-4a9a-b835-687ccc17feee nodeName:}" failed. No retries permitted until 2026-09-10 17:12:32.327157594 +0000 UTC m=+37.547281529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls") pod "dns-default-rh8lm" (UID: "10b2638b-880e-4a9a-b835-687ccc17feee") : secret "dns-default-metrics-tls" not found
Sep 10 17:12:31.497748 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.497626    2556 generic.go:356] "Generic (PLEG): container finished" podID="37b41989-8f5c-495f-91d6-faa4f75c2e97" containerID="65365a8cd5de2aeef6e8c5811f2344fe21a29071aeb1763d2e896a94fdb2abb8" exitCode=0
Sep 10 17:12:31.497748 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.497706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerDied","Data":"65365a8cd5de2aeef6e8c5811f2344fe21a29071aeb1763d2e896a94fdb2abb8"}
Sep 10 17:12:31.498937 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.498918    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7mbmg" event={"ID":"c7766669-96fc-4947-b998-827e01940bc8","Type":"ContainerStarted","Data":"732b9651b6b18fc299fe11536556581eff2db4f1433d4fdd26ac0c070f40bf80"}
Sep 10 17:12:31.543167 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.543139    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5"]
Sep 10 17:12:31.546963 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:31.546932    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba152442_477b_4c0f_959e_690826579bd1.slice/crio-2da627962101a03a47fc01e17256581c9bd66e0e3b7901b72b7d9ff73c405ee2 WatchSource:0}: Error finding container 2da627962101a03a47fc01e17256581c9bd66e0e3b7901b72b7d9ff73c405ee2: Status 404 returned error can't find the container with id 2da627962101a03a47fc01e17256581c9bd66e0e3b7901b72b7d9ff73c405ee2
Sep 10 17:12:31.931218 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.931138    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-44wdl"]
Sep 10 17:12:31.948497 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.948464    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-44wdl"]
Sep 10 17:12:31.948686 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.948582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:31.951252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.950915    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 17:12:31.951252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.951033    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 17:12:31.951252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.951148    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 17:12:31.951456 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.951258    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2k9xc\""
Sep 10 17:12:31.951456 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:31.951335    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 17:12:32.034090 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.034054    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jrk\" (UniqueName: \"kubernetes.io/projected/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-api-access-95jrk\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.034277 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.034231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d5c734b-d5a6-43ba-9da7-f298828e310c-data-volume\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.034338 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.034279    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.034338 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.034309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.034436 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.034359    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d5c734b-d5a6-43ba-9da7-f298828e310c-crio-socket\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.134889 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.134847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d5c734b-d5a6-43ba-9da7-f298828e310c-crio-socket\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.135062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.134895    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jrk\" (UniqueName: \"kubernetes.io/projected/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-api-access-95jrk\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.135062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.134991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d5c734b-d5a6-43ba-9da7-f298828e310c-data-volume\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.135062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.135024    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.135062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.135051    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.135244 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.135144    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d5c734b-d5a6-43ba-9da7-f298828e310c-crio-socket\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.135244 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.135160    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:32.135244 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.135189    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:12:32.135244 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.135224    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:32.63520434 +0000 UTC m=+37.855328266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:32.135244 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.135241    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:32.635231623 +0000 UTC m=+37.855355533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : configmap "kube-rbac-proxy" not found
Sep 10 17:12:32.135499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.135392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d5c734b-d5a6-43ba-9da7-f298828e310c-data-volume\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.144725 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.144691    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jrk\" (UniqueName: \"kubernetes.io/projected/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-api-access-95jrk\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.265872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.265555    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:12:32.265872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.265595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:12:32.265872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.265595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:12:32.268828 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.268525    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:12:32.268828 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.268591    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:12:32.268828 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.268641    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2g8g\""
Sep 10 17:12:32.269020 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.268919    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vgb7\""
Sep 10 17:12:32.269161 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.269146    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:12:32.269461 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.269433    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:12:32.336898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.336870    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:32.337320 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.337001    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:12:32.337320 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.337059    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls podName:10b2638b-880e-4a9a-b835-687ccc17feee nodeName:}" failed. No retries permitted until 2026-09-10 17:12:34.33704275 +0000 UTC m=+39.557166663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls") pod "dns-default-rh8lm" (UID: "10b2638b-880e-4a9a-b835-687ccc17feee") : secret "dns-default-metrics-tls" not found
Sep 10 17:12:32.503559 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.503521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5" event={"ID":"ba152442-477b-4c0f-959e-690826579bd1","Type":"ContainerStarted","Data":"2da627962101a03a47fc01e17256581c9bd66e0e3b7901b72b7d9ff73c405ee2"}
Sep 10 17:12:32.508061 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.507971    2556 generic.go:356] "Generic (PLEG): container finished" podID="37b41989-8f5c-495f-91d6-faa4f75c2e97" containerID="e002701eb9913d07af1e5214e87d923d6096de7328c91c517dba299f0cca0be3" exitCode=0
Sep 10 17:12:32.508061 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.508013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerDied","Data":"e002701eb9913d07af1e5214e87d923d6096de7328c91c517dba299f0cca0be3"}
Sep 10 17:12:32.639887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.639852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.640036 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:32.639900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:32.640036 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.639993    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:12:32.640151 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.640055    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:33.640035885 +0000 UTC m=+38.860159819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : configmap "kube-rbac-proxy" not found
Sep 10 17:12:32.640247 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.640231    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:32.640301 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:32.640275    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:33.640261833 +0000 UTC m=+38.860385744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:33.104620 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.104586    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d5vqd"]
Sep 10 17:12:33.117023 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.116989    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d5vqd"]
Sep 10 17:12:33.117181 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.117121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.119844 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.119821    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 17:12:33.119981 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.119930    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 17:12:33.120155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.120134    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f8kdz\""
Sep 10 17:12:33.120763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.120746    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 17:12:33.121179 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.121160    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 17:12:33.244204 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.244171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsvq\" (UniqueName: \"kubernetes.io/projected/2449b3f6-a838-4155-9f7e-af25316a56a7-kube-api-access-7qsvq\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.244350 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.244218    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2449b3f6-a838-4155-9f7e-af25316a56a7-signing-key\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.244408 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.244382    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2449b3f6-a838-4155-9f7e-af25316a56a7-signing-cabundle\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.345782 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.345748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2449b3f6-a838-4155-9f7e-af25316a56a7-signing-cabundle\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.346222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.345820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsvq\" (UniqueName: \"kubernetes.io/projected/2449b3f6-a838-4155-9f7e-af25316a56a7-kube-api-access-7qsvq\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.346222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.345854    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2449b3f6-a838-4155-9f7e-af25316a56a7-signing-key\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.346694 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.346620    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/2449b3f6-a838-4155-9f7e-af25316a56a7-signing-cabundle\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.349475 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.349446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/2449b3f6-a838-4155-9f7e-af25316a56a7-signing-key\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.358149 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.358090    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsvq\" (UniqueName: \"kubernetes.io/projected/2449b3f6-a838-4155-9f7e-af25316a56a7-kube-api-access-7qsvq\") pod \"service-ca-75d99c484-d5vqd\" (UID: \"2449b3f6-a838-4155-9f7e-af25316a56a7\") " pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.427992 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.427954    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-d5vqd"
Sep 10 17:12:33.515330 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.515296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" event={"ID":"37b41989-8f5c-495f-91d6-faa4f75c2e97","Type":"ContainerStarted","Data":"dfb0fefc8c8d66954fd336c087c091e560abb24b0ce5b66f2125ace1d74ddd47"}
Sep 10 17:12:33.538176 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.538133    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2gcq6" podStartSLOduration=7.34467195 podStartE2EDuration="37.538116608s" totalImagesPullingTime="30.193444658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:33.537623316 +0000 UTC m=+38.757747271" watchObservedRunningTime="2026-09-10 17:12:33.538116608 +0000 UTC m=+38.758240569"
Sep 10 17:12:33.647506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.647423    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:33.647506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:33.647462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:33.647748 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:33.647649    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:33.647808 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:33.647750    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:35.647720674 +0000 UTC m=+40.867844590 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:33.648067 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:33.648041    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:12:33.648209 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:33.648112    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:35.648094417 +0000 UTC m=+40.868218358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : configmap "kube-rbac-proxy" not found
Sep 10 17:12:34.353218 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.353062    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:34.353698 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:34.353239    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:12:34.353698 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:34.353307    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls podName:10b2638b-880e-4a9a-b835-687ccc17feee nodeName:}" failed. No retries permitted until 2026-09-10 17:12:38.353292108 +0000 UTC m=+43.573416019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls") pod "dns-default-rh8lm" (UID: "10b2638b-880e-4a9a-b835-687ccc17feee") : secret "dns-default-metrics-tls" not found
Sep 10 17:12:34.504296 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.504266    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-d5vqd"]
Sep 10 17:12:34.508849 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:34.508816    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2449b3f6_a838_4155_9f7e_af25316a56a7.slice/crio-1170654278e5ef884451960230d084604b60b9f65023d116f5111334a51b4ea7 WatchSource:0}: Error finding container 1170654278e5ef884451960230d084604b60b9f65023d116f5111334a51b4ea7: Status 404 returned error can't find the container with id 1170654278e5ef884451960230d084604b60b9f65023d116f5111334a51b4ea7
Sep 10 17:12:34.519522 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.519492    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7mbmg" event={"ID":"c7766669-96fc-4947-b998-827e01940bc8","Type":"ContainerStarted","Data":"81edfd89a1dbb1ea8e37cab2224265faa1eea6dab9fbb2710a7e06b430b53e68"}
Sep 10 17:12:34.520860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.520837    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d5vqd" event={"ID":"2449b3f6-a838-4155-9f7e-af25316a56a7","Type":"ContainerStarted","Data":"1170654278e5ef884451960230d084604b60b9f65023d116f5111334a51b4ea7"}
Sep 10 17:12:34.522354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.522332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5" event={"ID":"ba152442-477b-4c0f-959e-690826579bd1","Type":"ContainerStarted","Data":"10ff6444f49e2831d87f7021ef4ec23cc0faedb8bbebd74437c9db09294a63ac"}
Sep 10 17:12:34.545870 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.545830    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7mbmg" podStartSLOduration=1.599438464 podStartE2EDuration="4.545816141s" totalImagesPullingTime="2.946377677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:34.544919862 +0000 UTC m=+39.765043796" watchObservedRunningTime="2026-09-10 17:12:34.545816141 +0000 UTC m=+39.765940074"
Sep 10 17:12:34.897590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:34.897559    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxqp5_14bbd0d2-4568-41dc-ad8f-6129d10314f8/dns-node-resolver/0.log"
Sep 10 17:12:35.497474 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:35.497451    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2t5g6_9be1264a-5add-475d-9fe4-adcf3db28e24/node-ca/0.log"
Sep 10 17:12:35.526702 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:35.526648    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5" event={"ID":"ba152442-477b-4c0f-959e-690826579bd1","Type":"ContainerStarted","Data":"b9e397698e8a684c85385a62780a1b065d6d97f58d33b80d8a331d8cefc3ad8a"}
Sep 10 17:12:35.546915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:35.546851    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-xzfp5" podStartSLOduration=2.862307312 podStartE2EDuration="5.546829688s" totalImagesPullingTime="2.684522376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:35.546548167 +0000 UTC m=+40.766672112" watchObservedRunningTime="2026-09-10 17:12:35.546829688 +0000 UTC m=+40.766953621"
Sep 10 17:12:35.663760 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:35.663722    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:35.663997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:35.663774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:35.663997 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:35.663794    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:12:35.663997 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:35.663854    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.663840089 +0000 UTC m=+44.883964025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : configmap "kube-rbac-proxy" not found
Sep 10 17:12:35.663997 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:35.663920    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:35.663997 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:35.663982    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:39.663966272 +0000 UTC m=+44.884090186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:36.297945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:36.297912    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7mbmg_c7766669-96fc-4947-b998-827e01940bc8/serve-healthcheck-canary/0.log"
Sep 10 17:12:36.907017 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:36.906973    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xzfp5_ba152442-477b-4c0f-959e-690826579bd1/migrator/0.log"
Sep 10 17:12:37.098511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:37.098488    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xzfp5_ba152442-477b-4c0f-959e-690826579bd1/graceful-termination/0.log"
Sep 10 17:12:37.533020 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:37.532988    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-d5vqd" event={"ID":"2449b3f6-a838-4155-9f7e-af25316a56a7","Type":"ContainerStarted","Data":"601704a3ee92659c0e2f92e119ad12711d823fd3f33ba8cb7d5bb8cfb128c6bf"}
Sep 10 17:12:37.552432 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:37.552385    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-d5vqd" podStartSLOduration=2.0859933 podStartE2EDuration="4.552370622s" totalImagesPullingTime="2.466377322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:37.551922033 +0000 UTC m=+42.772045968" watchObservedRunningTime="2026-09-10 17:12:37.552370622 +0000 UTC m=+42.772494554"
Sep 10 17:12:38.382724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:38.382683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:38.383202 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:38.382798    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 17:12:38.383202 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:38.382881    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls podName:10b2638b-880e-4a9a-b835-687ccc17feee nodeName:}" failed. No retries permitted until 2026-09-10 17:12:46.382857913 +0000 UTC m=+51.602981828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls") pod "dns-default-rh8lm" (UID: "10b2638b-880e-4a9a-b835-687ccc17feee") : secret "dns-default-metrics-tls" not found
Sep 10 17:12:39.691949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:39.691903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:39.691949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:39.691945    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:39.692469 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:39.692034    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 17:12:39.692469 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:39.692104    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:47.692084143 +0000 UTC m=+52.912208068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : configmap "kube-rbac-proxy" not found
Sep 10 17:12:39.692469 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:39.692121    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:39.692469 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:39.692177    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:12:47.69216262 +0000 UTC m=+52.912286548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:46.440163 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:46.440114    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:46.443462 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:46.443434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10b2638b-880e-4a9a-b835-687ccc17feee-metrics-tls\") pod \"dns-default-rh8lm\" (UID: \"10b2638b-880e-4a9a-b835-687ccc17feee\") " pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:46.550607 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:46.550561    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:46.834225 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:46.834192    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rh8lm"]
Sep 10 17:12:46.854169 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:46.854136    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b2638b_880e_4a9a_b835_687ccc17feee.slice/crio-39d73b00566a617f2540a59094e2617ae481e468e4d1712b87dbb22138094999 WatchSource:0}: Error finding container 39d73b00566a617f2540a59094e2617ae481e468e4d1712b87dbb22138094999: Status 404 returned error can't find the container with id 39d73b00566a617f2540a59094e2617ae481e468e4d1712b87dbb22138094999
Sep 10 17:12:47.560907 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:47.560749    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rh8lm" event={"ID":"10b2638b-880e-4a9a-b835-687ccc17feee","Type":"ContainerStarted","Data":"39d73b00566a617f2540a59094e2617ae481e468e4d1712b87dbb22138094999"}
Sep 10 17:12:47.749108 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:47.749066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:47.749290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:47.749117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:47.749290 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:47.749258    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:47.749397 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:47.749327    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls podName:9d5c734b-d5a6-43ba-9da7-f298828e310c nodeName:}" failed. No retries permitted until 2026-09-10 17:13:03.749306795 +0000 UTC m=+68.969430709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44wdl" (UID: "9d5c734b-d5a6-43ba-9da7-f298828e310c") : secret "insights-runtime-extractor-tls" not found
Sep 10 17:12:47.749788 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:47.749765    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d5c734b-d5a6-43ba-9da7-f298828e310c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:12:48.567127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:48.567043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rh8lm" event={"ID":"10b2638b-880e-4a9a-b835-687ccc17feee","Type":"ContainerStarted","Data":"266067f12d83b9dbdb7b8ac01b0a87677ad5665221d6ed71fe901eb98e1964b3"}
Sep 10 17:12:49.571386 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:49.571345    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rh8lm" event={"ID":"10b2638b-880e-4a9a-b835-687ccc17feee","Type":"ContainerStarted","Data":"64c6f734895dbfcef5e095b1f5f9164937f3eaa7c1d25ce4c6464205a3867229"}
Sep 10 17:12:49.571878 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:49.571476    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:49.588168 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:49.588063    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rh8lm" podStartSLOduration=18.001725431 podStartE2EDuration="19.588045805s" totalImagesPullingTime="1.586320374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:49.587959646 +0000 UTC m=+54.808083580" watchObservedRunningTime="2026-09-10 17:12:49.588045805 +0000 UTC m=+54.808169738"
Sep 10 17:12:54.785804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.785771    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"]
Sep 10 17:12:54.814578 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.814546    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"]
Sep 10 17:12:54.814774 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.814539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:54.817457 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.817430    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 17:12:54.817576 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.817430    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4zjkk\""
Sep 10 17:12:54.818706 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.818688    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 17:12:54.818822 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.818727    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 17:12:54.818822 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.818727    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 17:12:54.881523 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.881490    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jtw2d"]
Sep 10 17:12:54.895748 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.895722    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drsm7\" (UniqueName: \"kubernetes.io/projected/4990069a-ba9a-4e5b-bb38-3d101776b916-kube-api-access-drsm7\") pod \"managed-serviceaccount-addon-agent-55dff6cd58-rsph8\" (UID: \"4990069a-ba9a-4e5b-bb38-3d101776b916\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:54.895872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.895768    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4990069a-ba9a-4e5b-bb38-3d101776b916-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55dff6cd58-rsph8\" (UID: \"4990069a-ba9a-4e5b-bb38-3d101776b916\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:54.904554 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.904532    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5987474868-bvs76"]
Sep 10 17:12:54.904691 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.904648    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:54.906986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.906971    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-skq29\""
Sep 10 17:12:54.907292 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.907275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 17:12:54.907521 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.907505    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 17:12:54.924746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.924729    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jtw2d"]
Sep 10 17:12:54.924746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.924758    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5987474868-bvs76"]
Sep 10 17:12:54.924901 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.924841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.927574 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.927557    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 17:12:54.927890 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.927870    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ppxr2\""
Sep 10 17:12:54.928372 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.928358    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 17:12:54.928646 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.928633    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 17:12:54.933261 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.933241    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 17:12:54.996436 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-registry-certificates\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996439    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9vl\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-kube-api-access-tf9vl\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996474    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bf26107-29a9-488c-8d83-f60f2cc0d540-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jtw2d\" (UID: \"5bf26107-29a9-488c-8d83-f60f2cc0d540\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:54.996585 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drsm7\" (UniqueName: \"kubernetes.io/projected/4990069a-ba9a-4e5b-bb38-3d101776b916-kube-api-access-drsm7\") pod \"managed-serviceaccount-addon-agent-55dff6cd58-rsph8\" (UID: \"4990069a-ba9a-4e5b-bb38-3d101776b916\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:54.996585 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996576    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-registry-tls\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996651 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996606    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4990069a-ba9a-4e5b-bb38-3d101776b916-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55dff6cd58-rsph8\" (UID: \"4990069a-ba9a-4e5b-bb38-3d101776b916\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:54.996651 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bf26107-29a9-488c-8d83-f60f2cc0d540-nginx-conf\") pod \"networking-console-plugin-79977f9844-jtw2d\" (UID: \"5bf26107-29a9-488c-8d83-f60f2cc0d540\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:54.996742 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996651    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-image-registry-private-configuration\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996742 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996683    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-installation-pull-secrets\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996742 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-bound-sa-token\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996742 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996735    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-ca-trust-extracted\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.996912 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.996794    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-trusted-ca\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:54.999113 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:54.999095    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4990069a-ba9a-4e5b-bb38-3d101776b916-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-55dff6cd58-rsph8\" (UID: \"4990069a-ba9a-4e5b-bb38-3d101776b916\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:55.009407 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.009385    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drsm7\" (UniqueName: \"kubernetes.io/projected/4990069a-ba9a-4e5b-bb38-3d101776b916-kube-api-access-drsm7\") pod \"managed-serviceaccount-addon-agent-55dff6cd58-rsph8\" (UID: \"4990069a-ba9a-4e5b-bb38-3d101776b916\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:55.097569 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bf26107-29a9-488c-8d83-f60f2cc0d540-nginx-conf\") pod \"networking-console-plugin-79977f9844-jtw2d\" (UID: \"5bf26107-29a9-488c-8d83-f60f2cc0d540\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:55.097569 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097544    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-image-registry-private-configuration\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097569 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097559    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-installation-pull-secrets\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097575    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-bound-sa-token\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097592    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-ca-trust-extracted\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097609    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-trusted-ca\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-registry-certificates\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097654    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9vl\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-kube-api-access-tf9vl\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097722    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bf26107-29a9-488c-8d83-f60f2cc0d540-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jtw2d\" (UID: \"5bf26107-29a9-488c-8d83-f60f2cc0d540\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:55.097800 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.097771    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-registry-tls\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.098127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.098049    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-ca-trust-extracted\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.098418 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.098392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bf26107-29a9-488c-8d83-f60f2cc0d540-nginx-conf\") pod \"networking-console-plugin-79977f9844-jtw2d\" (UID: \"5bf26107-29a9-488c-8d83-f60f2cc0d540\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:55.098626 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.098600    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-trusted-ca\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.098848 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.098819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-registry-certificates\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.100024 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.099994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-image-registry-private-configuration\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.100135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.100118    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-installation-pull-secrets\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.100243 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.100223    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-registry-tls\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.100377 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.100357    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bf26107-29a9-488c-8d83-f60f2cc0d540-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-jtw2d\" (UID: \"5bf26107-29a9-488c-8d83-f60f2cc0d540\") " pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:55.105649 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.105610    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9vl\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-kube-api-access-tf9vl\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.105772 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.105658    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e-bound-sa-token\") pod \"image-registry-5987474868-bvs76\" (UID: \"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e\") " pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.137624 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.137606    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 17:12:55.148278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.148259    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-4zjkk\""
Sep 10 17:12:55.156256 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.156238    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"
Sep 10 17:12:55.216034 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.216009    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-skq29\""
Sep 10 17:12:55.224021 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.223786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d"
Sep 10 17:12:55.235010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.234989    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ppxr2\""
Sep 10 17:12:55.243093 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.243071    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:55.459328 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.459256    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8"]
Sep 10 17:12:55.462264 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:55.462240    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4990069a_ba9a_4e5b_bb38_3d101776b916.slice/crio-c95568fc1a3b561ef067c5a958a95003492df40f3b577ac7b6ed16de23bfac77 WatchSource:0}: Error finding container c95568fc1a3b561ef067c5a958a95003492df40f3b577ac7b6ed16de23bfac77: Status 404 returned error can't find the container with id c95568fc1a3b561ef067c5a958a95003492df40f3b577ac7b6ed16de23bfac77
Sep 10 17:12:55.518378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.518358    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-jtw2d"]
Sep 10 17:12:55.522138 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:55.522115    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf26107_29a9_488c_8d83_f60f2cc0d540.slice/crio-45ea75feb2e7a47aeeb0dfb164edb43a1f5acd95dcd3e08c5571d4b2ae2a4918 WatchSource:0}: Error finding container 45ea75feb2e7a47aeeb0dfb164edb43a1f5acd95dcd3e08c5571d4b2ae2a4918: Status 404 returned error can't find the container with id 45ea75feb2e7a47aeeb0dfb164edb43a1f5acd95dcd3e08c5571d4b2ae2a4918
Sep 10 17:12:55.562238 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.562215    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5987474868-bvs76"]
Sep 10 17:12:55.564153 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:12:55.564135    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7f7c1e_58f8_4634_bcbc_fdf08e55e41e.slice/crio-3ca9e2a97b627b783d6eb3c8406b2689eb846e46c02d2b5e821344b8c3ae479a WatchSource:0}: Error finding container 3ca9e2a97b627b783d6eb3c8406b2689eb846e46c02d2b5e821344b8c3ae479a: Status 404 returned error can't find the container with id 3ca9e2a97b627b783d6eb3c8406b2689eb846e46c02d2b5e821344b8c3ae479a
Sep 10 17:12:55.591748 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.591724    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8" event={"ID":"4990069a-ba9a-4e5b-bb38-3d101776b916","Type":"ContainerStarted","Data":"c95568fc1a3b561ef067c5a958a95003492df40f3b577ac7b6ed16de23bfac77"}
Sep 10 17:12:55.592784 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.592758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5987474868-bvs76" event={"ID":"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e","Type":"ContainerStarted","Data":"3ca9e2a97b627b783d6eb3c8406b2689eb846e46c02d2b5e821344b8c3ae479a"}
Sep 10 17:12:55.593756 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:55.593738    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d" event={"ID":"5bf26107-29a9-488c-8d83-f60f2cc0d540","Type":"ContainerStarted","Data":"45ea75feb2e7a47aeeb0dfb164edb43a1f5acd95dcd3e08c5571d4b2ae2a4918"}
Sep 10 17:12:56.498379 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:56.497927    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wxqt9"
Sep 10 17:12:56.603676 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:56.603622    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5987474868-bvs76" event={"ID":"3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e","Type":"ContainerStarted","Data":"9fada28993f54d1f3e98b0532c443d2626ee620632fda948e8a8ccb7c8c23de9"}
Sep 10 17:12:56.603867 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:56.603846    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:12:56.638397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:56.638140    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5987474868-bvs76" podStartSLOduration=2.638118013 podStartE2EDuration="2.638118013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:56.636909941 +0000 UTC m=+61.857033875" watchObservedRunningTime="2026-09-10 17:12:56.638118013 +0000 UTC m=+61.858241944"
Sep 10 17:12:57.611418 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:57.611378    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d" event={"ID":"5bf26107-29a9-488c-8d83-f60f2cc0d540","Type":"ContainerStarted","Data":"ab84c34646892bcbb2b365fb4c71893a21423f1afbb0c22d98edd8274c3c76d7"}
Sep 10 17:12:57.634895 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:57.634847    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-jtw2d" podStartSLOduration=2.279648312 podStartE2EDuration="3.634833412s" totalImagesPullingTime="1.3551851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:57.629715634 +0000 UTC m=+62.849839567" watchObservedRunningTime="2026-09-10 17:12:57.634833412 +0000 UTC m=+62.854957345"
Sep 10 17:12:58.617273 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:58.617239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8" event={"ID":"4990069a-ba9a-4e5b-bb38-3d101776b916","Type":"ContainerStarted","Data":"9dbc8acf67a2071d4bc68b1dbd75348c69b47c4d44b948d83eb289c7d9a5ba2b"}
Sep 10 17:12:58.634126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:58.634080    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-55dff6cd58-rsph8" podStartSLOduration=2.105476533 podStartE2EDuration="4.634066081s" totalImagesPullingTime="2.528589548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:12:58.632804088 +0000 UTC m=+63.852928020" watchObservedRunningTime="2026-09-10 17:12:58.634066081 +0000 UTC m=+63.854190014"
Sep 10 17:12:59.577700 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.577652    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rh8lm"
Sep 10 17:12:59.704492 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.704457    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"]
Sep 10 17:12:59.706822 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.706806    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:12:59.709110 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.709083    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6klk7\""
Sep 10 17:12:59.709247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.709119    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 17:12:59.716157 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.716134    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"]
Sep 10 17:12:59.730859 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.730830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0fb6ec77-0a92-4946-a40c-5a35b6a14649-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8hsfl\" (UID: \"0fb6ec77-0a92-4946-a40c-5a35b6a14649\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:12:59.832053 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:12:59.831965    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0fb6ec77-0a92-4946-a40c-5a35b6a14649-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8hsfl\" (UID: \"0fb6ec77-0a92-4946-a40c-5a35b6a14649\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:12:59.832198 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:59.832087    2556 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 17:12:59.832198 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:12:59.832141    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fb6ec77-0a92-4946-a40c-5a35b6a14649-tls-certificates podName:0fb6ec77-0a92-4946-a40c-5a35b6a14649 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:00.332126926 +0000 UTC m=+65.552250836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/0fb6ec77-0a92-4946-a40c-5a35b6a14649-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-8hsfl" (UID: "0fb6ec77-0a92-4946-a40c-5a35b6a14649") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 10 17:13:00.335240 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:00.335199    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0fb6ec77-0a92-4946-a40c-5a35b6a14649-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8hsfl\" (UID: \"0fb6ec77-0a92-4946-a40c-5a35b6a14649\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:13:00.337797 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:00.337759    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0fb6ec77-0a92-4946-a40c-5a35b6a14649-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-8hsfl\" (UID: \"0fb6ec77-0a92-4946-a40c-5a35b6a14649\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:13:00.616204 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:00.616100    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:13:00.881516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:00.881467    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"]
Sep 10 17:13:00.883866 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:00.883837    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb6ec77_0a92_4946_a40c_5a35b6a14649.slice/crio-6b5d871546cb6179a98d13f3cc7e090ba8734ac1ba25cf29f44491e271b961fa WatchSource:0}: Error finding container 6b5d871546cb6179a98d13f3cc7e090ba8734ac1ba25cf29f44491e271b961fa: Status 404 returned error can't find the container with id 6b5d871546cb6179a98d13f3cc7e090ba8734ac1ba25cf29f44491e271b961fa
Sep 10 17:13:01.040292 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.040256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:13:01.040474 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.040311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:13:01.040474 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.040339    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:13:01.042990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.042972    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 17:13:01.044130 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.044116    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 17:13:01.044175 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.044156    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 17:13:01.053365 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.053338    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29c79ac8-60f9-434a-9687-28a323a15418-original-pull-secret\") pod \"global-pull-secret-syncer-6fwnl\" (UID: \"29c79ac8-60f9-434a-9687-28a323a15418\") " pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:13:01.053504 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.053406    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcecaa29-0662-4242-81b6-94bf162b78fa-metrics-certs\") pod \"network-metrics-daemon-pz5km\" (UID: \"dcecaa29-0662-4242-81b6-94bf162b78fa\") " pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:13:01.053565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.053511    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 17:13:01.063232 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.063213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv87q\" (UniqueName: \"kubernetes.io/projected/14070417-5047-4893-9c28-c96bbccec68b-kube-api-access-sv87q\") pod \"network-check-target-wxqdz\" (UID: \"14070417-5047-4893-9c28-c96bbccec68b\") " pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:13:01.081549 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.081532    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-f2g8g\""
Sep 10 17:13:01.089595 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.089580    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:13:01.089931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.089916    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8vgb7\""
Sep 10 17:13:01.094816 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.094799    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6fwnl"
Sep 10 17:13:01.097915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.097902    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pz5km"
Sep 10 17:13:01.394224 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.394201    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wxqdz"]
Sep 10 17:13:01.396503 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:01.396477    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14070417_5047_4893_9c28_c96bbccec68b.slice/crio-b4d786f009c2c01eb48f979b2b657e5bf0476c8b780b1445cbf038fbbe9b97ba WatchSource:0}: Error finding container b4d786f009c2c01eb48f979b2b657e5bf0476c8b780b1445cbf038fbbe9b97ba: Status 404 returned error can't find the container with id b4d786f009c2c01eb48f979b2b657e5bf0476c8b780b1445cbf038fbbe9b97ba
Sep 10 17:13:01.416817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.416765    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6fwnl"]
Sep 10 17:13:01.419120 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:01.419098    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c79ac8_60f9_434a_9687_28a323a15418.slice/crio-9e64b03689bc79c87d4e566e30859ad260b51aaf3c3447f026dc901bdb9db8f7 WatchSource:0}: Error finding container 9e64b03689bc79c87d4e566e30859ad260b51aaf3c3447f026dc901bdb9db8f7: Status 404 returned error can't find the container with id 9e64b03689bc79c87d4e566e30859ad260b51aaf3c3447f026dc901bdb9db8f7
Sep 10 17:13:01.445957 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.445874    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pz5km"]
Sep 10 17:13:01.448477 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:01.448452    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcecaa29_0662_4242_81b6_94bf162b78fa.slice/crio-2eecde6a6ea906fac85b8e1e1fafd075e9a0263495ee79d06334f42caee6b634 WatchSource:0}: Error finding container 2eecde6a6ea906fac85b8e1e1fafd075e9a0263495ee79d06334f42caee6b634: Status 404 returned error can't find the container with id 2eecde6a6ea906fac85b8e1e1fafd075e9a0263495ee79d06334f42caee6b634
Sep 10 17:13:01.630622 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.627931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wxqdz" event={"ID":"14070417-5047-4893-9c28-c96bbccec68b","Type":"ContainerStarted","Data":"b4d786f009c2c01eb48f979b2b657e5bf0476c8b780b1445cbf038fbbe9b97ba"}
Sep 10 17:13:01.631921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.631890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pz5km" event={"ID":"dcecaa29-0662-4242-81b6-94bf162b78fa","Type":"ContainerStarted","Data":"2eecde6a6ea906fac85b8e1e1fafd075e9a0263495ee79d06334f42caee6b634"}
Sep 10 17:13:01.632997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.632975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl" event={"ID":"0fb6ec77-0a92-4946-a40c-5a35b6a14649","Type":"ContainerStarted","Data":"6b5d871546cb6179a98d13f3cc7e090ba8734ac1ba25cf29f44491e271b961fa"}
Sep 10 17:13:01.634075 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:01.634057    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6fwnl" event={"ID":"29c79ac8-60f9-434a-9687-28a323a15418","Type":"ContainerStarted","Data":"9e64b03689bc79c87d4e566e30859ad260b51aaf3c3447f026dc901bdb9db8f7"}
Sep 10 17:13:03.643704 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:03.643600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl" event={"ID":"0fb6ec77-0a92-4946-a40c-5a35b6a14649","Type":"ContainerStarted","Data":"1ab0dc0c44dac67d17f08940885d8de16e3e8a94de94a2ff6d0aa83d8f043d74"}
Sep 10 17:13:03.644154 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:03.643778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:13:03.649744 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:03.649717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl"
Sep 10 17:13:03.660010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:03.659957    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-8hsfl" podStartSLOduration=2.185622793 podStartE2EDuration="4.659938978s" totalImagesPullingTime="2.474316185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:03.659258896 +0000 UTC m=+68.879382830" watchObservedRunningTime="2026-09-10 17:13:03.659938978 +0000 UTC m=+68.880062912"
Sep 10 17:13:03.761528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:03.761432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:13:03.764240 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:03.764213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d5c734b-d5a6-43ba-9da7-f298828e310c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44wdl\" (UID: \"9d5c734b-d5a6-43ba-9da7-f298828e310c\") " pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:13:04.062829 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:04.062603    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2k9xc\""
Sep 10 17:13:04.070652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:04.070616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-44wdl"
Sep 10 17:13:04.648603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:04.648563    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pz5km" event={"ID":"dcecaa29-0662-4242-81b6-94bf162b78fa","Type":"ContainerStarted","Data":"27b85685769cc9508279f37bd796ed415e3d8212214aaae34e6bf65d52ea62c4"}
Sep 10 17:13:07.271948 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.271920    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-44wdl"]
Sep 10 17:13:07.273737 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:07.273705    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5c734b_d5a6_43ba_9da7_f298828e310c.slice/crio-62d1d1900bbbfceb55779d141f948193c606fb94da0723a140eb195099e6b4a2 WatchSource:0}: Error finding container 62d1d1900bbbfceb55779d141f948193c606fb94da0723a140eb195099e6b4a2: Status 404 returned error can't find the container with id 62d1d1900bbbfceb55779d141f948193c606fb94da0723a140eb195099e6b4a2
Sep 10 17:13:07.661346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.661261    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wxqdz" event={"ID":"14070417-5047-4893-9c28-c96bbccec68b","Type":"ContainerStarted","Data":"a092ef31da0cb6f3e892669d150a20c89233a161da35e5c62d149a72bf72fea4"}
Sep 10 17:13:07.661503 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.661381    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:13:07.662856 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.662829    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44wdl" event={"ID":"9d5c734b-d5a6-43ba-9da7-f298828e310c","Type":"ContainerStarted","Data":"92bb269a31a2df1c9bd32c3eaf6baacc0e17e62d5f4fa3101dd1fa2ec037b48e"}
Sep 10 17:13:07.662962 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.662885    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44wdl" event={"ID":"9d5c734b-d5a6-43ba-9da7-f298828e310c","Type":"ContainerStarted","Data":"62d1d1900bbbfceb55779d141f948193c606fb94da0723a140eb195099e6b4a2"}
Sep 10 17:13:07.664733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.664704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pz5km" event={"ID":"dcecaa29-0662-4242-81b6-94bf162b78fa","Type":"ContainerStarted","Data":"5c71b360977efd621ac5900b3e734755e5b907364b6780b6f0a19d65f1175c1b"}
Sep 10 17:13:07.666054 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.666034    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6fwnl" event={"ID":"29c79ac8-60f9-434a-9687-28a323a15418","Type":"ContainerStarted","Data":"fa8e311cd550392d2ffbadb4776f8ea72e0edf3428e901639ea52ba5d97e5fc4"}
Sep 10 17:13:07.677304 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.677261    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wxqdz" podStartSLOduration=66.076882718 podStartE2EDuration="1m11.677240395s" totalImagesPullingTime="5.600357677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:07.676840076 +0000 UTC m=+72.896964008" watchObservedRunningTime="2026-09-10 17:13:07.677240395 +0000 UTC m=+72.897364333"
Sep 10 17:13:07.693324 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.693291    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6fwnl" podStartSLOduration=66.105301149 podStartE2EDuration="1m11.693279052s" totalImagesPullingTime="5.587977903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:07.692388377 +0000 UTC m=+72.912512312" watchObservedRunningTime="2026-09-10 17:13:07.693279052 +0000 UTC m=+72.913402984"
Sep 10 17:13:07.707062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:07.707020    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pz5km" podStartSLOduration=69.47064498 podStartE2EDuration="1m11.707006949s" totalImagesPullingTime="2.236361969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:07.706881354 +0000 UTC m=+72.927005288" watchObservedRunningTime="2026-09-10 17:13:07.707006949 +0000 UTC m=+72.927130883"
Sep 10 17:13:08.672763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:08.672720    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44wdl" event={"ID":"9d5c734b-d5a6-43ba-9da7-f298828e310c","Type":"ContainerStarted","Data":"3abdc0202c53be98af0d68748bbfe984485e3586d6612aef27e946168d2dd5f5"}
Sep 10 17:13:09.678166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:09.678076    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44wdl" event={"ID":"9d5c734b-d5a6-43ba-9da7-f298828e310c","Type":"ContainerStarted","Data":"0e0d1df55fa33a22b2f196fd31a939cc9a28a1346f9c642293fcce455d1ed653"}
Sep 10 17:13:09.700681 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:09.698768    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-44wdl" podStartSLOduration=36.933478959 podStartE2EDuration="38.69874948s" totalImagesPullingTime="1.765270521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:12:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:09.698214334 +0000 UTC m=+74.918338267" watchObservedRunningTime="2026-09-10 17:13:09.69874948 +0000 UTC m=+74.918873414"
Sep 10 17:13:10.130108 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.130067    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"]
Sep 10 17:13:10.132771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.132753    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.135101 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.135075    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 17:13:10.135372 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.135356    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 17:13:10.135686 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.135644    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 17:13:10.136107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.136045    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 17:13:10.136107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.136052    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-g2pbw\""
Sep 10 17:13:10.136238 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.136089    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 17:13:10.146460 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.146436    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"]
Sep 10 17:13:10.207192 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.207153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.207360 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.207302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.207413 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.207355    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr49v\" (UniqueName: \"kubernetes.io/projected/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-kube-api-access-wr49v\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.207457 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.207419    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.248000 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.247970    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gpztw"]
Sep 10 17:13:10.250391 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.250367    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.252855 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.252823    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 17:13:10.252941 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.252837    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 17:13:10.253413 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.253396    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-279hs\""
Sep 10 17:13:10.253508 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.253472    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 17:13:10.308455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.308640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308463    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-sys\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.308640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308499    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-tls\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308521    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308640 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:13:10.308601    2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 17:13:10.308640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308621    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-wtmp\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ldv\" (UniqueName: \"kubernetes.io/projected/599f330f-e7b2-4cc2-998a-8a3de3084b88-kube-api-access-95ldv\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:13:10.308695    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-tls podName:0352af2d-c1da-4b0a-bff1-5f54daaa0a17 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:10.808658355 +0000 UTC m=+76.028782271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-vk9ms" (UID: "0352af2d-c1da-4b0a-bff1-5f54daaa0a17") : secret "openshift-state-metrics-tls" not found
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-root\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308899    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/599f330f-e7b2-4cc2-998a-8a3de3084b88-metrics-client-ca\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr49v\" (UniqueName: \"kubernetes.io/projected/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-kube-api-access-wr49v\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.308971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.308954    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-textfile\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.309364 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.309274    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.311320 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.311300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.324191 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.324167    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr49v\" (UniqueName: \"kubernetes.io/projected/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-kube-api-access-wr49v\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.409472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409379    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-root\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409418    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/599f330f-e7b2-4cc2-998a-8a3de3084b88-metrics-client-ca\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409438    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-textfile\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-sys\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409490    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-tls\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-root\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409543    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-sys\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409519    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409625    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-wtmp\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409649    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ldv\" (UniqueName: \"kubernetes.io/projected/599f330f-e7b2-4cc2-998a-8a3de3084b88-kube-api-access-95ldv\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.409789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.410091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409813    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-wtmp\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.410091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.409931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-textfile\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.410216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.410192    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/599f330f-e7b2-4cc2-998a-8a3de3084b88-metrics-client-ca\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.410287 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.410218    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-accelerators-collector-config\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.411981 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.411953    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-tls\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.412112 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.412088    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/599f330f-e7b2-4cc2-998a-8a3de3084b88-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.425013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.424986    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ldv\" (UniqueName: \"kubernetes.io/projected/599f330f-e7b2-4cc2-998a-8a3de3084b88-kube-api-access-95ldv\") pod \"node-exporter-gpztw\" (UID: \"599f330f-e7b2-4cc2-998a-8a3de3084b88\") " pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.562674 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.562631    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gpztw"
Sep 10 17:13:10.572955 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:10.572927    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599f330f_e7b2_4cc2_998a_8a3de3084b88.slice/crio-4db239b2ec4b20854e73297ff6a7dab91d9d163c97d9333f839686435d3153f7 WatchSource:0}: Error finding container 4db239b2ec4b20854e73297ff6a7dab91d9d163c97d9333f839686435d3153f7: Status 404 returned error can't find the container with id 4db239b2ec4b20854e73297ff6a7dab91d9d163c97d9333f839686435d3153f7
Sep 10 17:13:10.682607 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.682521    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpztw" event={"ID":"599f330f-e7b2-4cc2-998a-8a3de3084b88","Type":"ContainerStarted","Data":"4db239b2ec4b20854e73297ff6a7dab91d9d163c97d9333f839686435d3153f7"}
Sep 10 17:13:10.814235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.814201    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:10.816429 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:10.816409    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0352af2d-c1da-4b0a-bff1-5f54daaa0a17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-vk9ms\" (UID: \"0352af2d-c1da-4b0a-bff1-5f54daaa0a17\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:11.043249 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.043217    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"
Sep 10 17:13:11.312329 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.312217    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms"]
Sep 10 17:13:11.316267 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:11.316233    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0352af2d_c1da_4b0a_bff1_5f54daaa0a17.slice/crio-0fcafcdebd248712f9f8c538fd69f9d6ecc9d0b47bb06ef67d9fc6bb3ac348a8 WatchSource:0}: Error finding container 0fcafcdebd248712f9f8c538fd69f9d6ecc9d0b47bb06ef67d9fc6bb3ac348a8: Status 404 returned error can't find the container with id 0fcafcdebd248712f9f8c538fd69f9d6ecc9d0b47bb06ef67d9fc6bb3ac348a8
Sep 10 17:13:11.342829 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.342803    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:13:11.346346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.346327    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.348737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.348714    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350288    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350309    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350329    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-69fhq\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350339    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350347    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350291    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350395    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 17:13:11.350459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.350350    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 17:13:11.351034 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.351016    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 17:13:11.367703 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.367679    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:13:11.419261 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419205    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419261 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419386    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-web-config\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419443    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419471    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419571 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419571 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-config-out\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419645 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g972f\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-kube-api-access-g972f\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419645 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419645 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419628    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.419792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.419756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-config-volume\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520574 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520542    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-config-out\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520574 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g972f\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-kube-api-access-g972f\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520598    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520790    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520832    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-config-volume\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.520945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520919    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.521107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.520968    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-web-config\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.521107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.521033    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.521107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.521057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.521107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.521088    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.521107 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:13:11.521103    2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 17:13:11.521341 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:13:11.521176    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls podName:8343e387-06c0-4329-92e4-bf5dff4e4192 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:12.021153323 +0000 UTC m=+77.241277241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192") : secret "alertmanager-main-tls" not found
Sep 10 17:13:11.521732 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.521698    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.522000 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.521971    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.522952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.522927    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.523688 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.523636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-config-out\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.526029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.525569    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.526029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.525727    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-config-volume\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.526029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.525932    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.526029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.525984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.526029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.526032    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.527075 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.527026    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-web-config\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.527173 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.527112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.537174 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.537149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g972f\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-kube-api-access-g972f\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:11.688560 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.688126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms" event={"ID":"0352af2d-c1da-4b0a-bff1-5f54daaa0a17","Type":"ContainerStarted","Data":"52d930bb23c1524c5aa23382403bdc0c8e62f7a0012984260050c62b291870e5"}
Sep 10 17:13:11.688560 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.688224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms" event={"ID":"0352af2d-c1da-4b0a-bff1-5f54daaa0a17","Type":"ContainerStarted","Data":"f7d1c958ce9ed1ee3c36233e5277b1236b34703ecee3226066d27b37ce810007"}
Sep 10 17:13:11.688560 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:11.688252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms" event={"ID":"0352af2d-c1da-4b0a-bff1-5f54daaa0a17","Type":"ContainerStarted","Data":"0fcafcdebd248712f9f8c538fd69f9d6ecc9d0b47bb06ef67d9fc6bb3ac348a8"}
Sep 10 17:13:12.024822 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.024785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:12.027592 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.027563    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:12.245975 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.245941    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"]
Sep 10 17:13:12.248834 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.248804    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.251951 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.251924    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 17:13:12.252060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.251924    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-771u8t8vv6iu2\""
Sep 10 17:13:12.252060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.252026    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 17:13:12.252400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.252378    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sqg9c\""
Sep 10 17:13:12.252524 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.252383    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 17:13:12.252524 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.252431    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 17:13:12.253098 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.253080    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 17:13:12.269163 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.269138    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"]
Sep 10 17:13:12.275527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.275463    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:13:12.327740 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5p5l\" (UniqueName: \"kubernetes.io/projected/d6db4dea-e1d1-425c-9486-50297d78e145-kube-api-access-z5p5l\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.327875 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-tls\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.327875 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.327875 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327808    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.327875 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-grpc-tls\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.328045 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327909    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6db4dea-e1d1-425c-9486-50297d78e145-metrics-client-ca\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.328045 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327938    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.328045 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.327963    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429379 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6db4dea-e1d1-425c-9486-50297d78e145-metrics-client-ca\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429487 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429487 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429442    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429487 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5p5l\" (UniqueName: \"kubernetes.io/projected/d6db4dea-e1d1-425c-9486-50297d78e145-kube-api-access-z5p5l\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-tls\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429532    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.429643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.429610    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-grpc-tls\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.432492 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.432460    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6db4dea-e1d1-425c-9486-50297d78e145-metrics-client-ca\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.436010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.435980    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.436119 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.436094    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.436218 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.436199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-tls\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.436994 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.436954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.438365 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.437961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-grpc-tls\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.438365 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.438329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6db4dea-e1d1-425c-9486-50297d78e145-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.446779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.446736    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5p5l\" (UniqueName: \"kubernetes.io/projected/d6db4dea-e1d1-425c-9486-50297d78e145-kube-api-access-z5p5l\") pod \"thanos-querier-7b49f6574f-kxbmr\" (UID: \"d6db4dea-e1d1-425c-9486-50297d78e145\") " pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.559768 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.559685    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:12.651464 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.651438    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:13:12.652917 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:12.652891    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8343e387_06c0_4329_92e4_bf5dff4e4192.slice/crio-713384f066057b9e6a6afda398e301b81c7efc112bab5f19ed71d06f880812cd WatchSource:0}: Error finding container 713384f066057b9e6a6afda398e301b81c7efc112bab5f19ed71d06f880812cd: Status 404 returned error can't find the container with id 713384f066057b9e6a6afda398e301b81c7efc112bab5f19ed71d06f880812cd
Sep 10 17:13:12.693010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.692977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"713384f066057b9e6a6afda398e301b81c7efc112bab5f19ed71d06f880812cd"}
Sep 10 17:13:12.694623 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.694598    2556 generic.go:356] "Generic (PLEG): container finished" podID="599f330f-e7b2-4cc2-998a-8a3de3084b88" containerID="29cd05b423d5e553ef2bb24bf460af215a1b9692602b241009ce0cd0e14fe46d" exitCode=0
Sep 10 17:13:12.694736 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.694694    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpztw" event={"ID":"599f330f-e7b2-4cc2-998a-8a3de3084b88","Type":"ContainerDied","Data":"29cd05b423d5e553ef2bb24bf460af215a1b9692602b241009ce0cd0e14fe46d"}
Sep 10 17:13:12.696995 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.696931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms" event={"ID":"0352af2d-c1da-4b0a-bff1-5f54daaa0a17","Type":"ContainerStarted","Data":"030981a7b5247c6861fd2a510fb7672b68874239d002fd6bde67e7dc7c57b824"}
Sep 10 17:13:12.736096 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.736056    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-vk9ms" podStartSLOduration=1.777038112 podStartE2EDuration="2.736042211s" totalImagesPullingTime="959.004099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:12.734989932 +0000 UTC m=+77.955113876" watchObservedRunningTime="2026-09-10 17:13:12.736042211 +0000 UTC m=+77.956166143"
Sep 10 17:13:12.830568 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:12.830542    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"]
Sep 10 17:13:12.832098 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:12.832062    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6db4dea_e1d1_425c_9486_50297d78e145.slice/crio-dea2fcf9eec7de4bb7e756daa8db9d3059a0659447d992f3e7039801e1ff7251 WatchSource:0}: Error finding container dea2fcf9eec7de4bb7e756daa8db9d3059a0659447d992f3e7039801e1ff7251: Status 404 returned error can't find the container with id dea2fcf9eec7de4bb7e756daa8db9d3059a0659447d992f3e7039801e1ff7251
Sep 10 17:13:13.704930 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:13.704874    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpztw" event={"ID":"599f330f-e7b2-4cc2-998a-8a3de3084b88","Type":"ContainerStarted","Data":"a07c9064f39d04dffc85a894f9f6d1da3aa5f025a2fb3ddaf9a9cb34e926985c"}
Sep 10 17:13:13.705395 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:13.704947    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gpztw" event={"ID":"599f330f-e7b2-4cc2-998a-8a3de3084b88","Type":"ContainerStarted","Data":"475945e791c4371605fa8496311df13ee60f2d69e1373f5803d41eba59dc271f"}
Sep 10 17:13:13.706442 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:13.706413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"dea2fcf9eec7de4bb7e756daa8db9d3059a0659447d992f3e7039801e1ff7251"}
Sep 10 17:13:13.729120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:13.729070    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gpztw" podStartSLOduration=2.551949929 podStartE2EDuration="3.729055434s" totalImagesPullingTime="1.115158653s" totalInitContainerRuntime="61.946852ms" isStatefulPod=true podCreationTimestamp="2026-09-10 17:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:13.72690032 +0000 UTC m=+78.947024286" watchObservedRunningTime="2026-09-10 17:13:13.729055434 +0000 UTC m=+78.949179366"
Sep 10 17:13:14.712068 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.712029    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a" exitCode=0
Sep 10 17:13:14.712548 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.712116    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"}
Sep 10 17:13:14.909306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.909268    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"]
Sep 10 17:13:14.911931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.911913    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:14.914208 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.914185    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 17:13:14.914296 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.914193    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ljb29\""
Sep 10 17:13:14.922220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.922162    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"]
Sep 10 17:13:14.952681 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.952633    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d04454e-e78a-4736-86fd-9cf2cc563221-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:14.952810 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:14.952705    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5lhz\" (UniqueName: \"kubernetes.io/projected/0d04454e-e78a-4736-86fd-9cf2cc563221-kube-api-access-j5lhz\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:15.053411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.053382    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lhz\" (UniqueName: \"kubernetes.io/projected/0d04454e-e78a-4736-86fd-9cf2cc563221-kube-api-access-j5lhz\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:15.053497 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.053481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d04454e-e78a-4736-86fd-9cf2cc563221-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:15.053602 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:13:15.053589    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 10 17:13:15.053678 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:13:15.053650    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d04454e-e78a-4736-86fd-9cf2cc563221-monitoring-plugin-cert podName:0d04454e-e78a-4736-86fd-9cf2cc563221 nodeName:}" failed. No retries permitted until 2026-09-10 17:13:15.553630806 +0000 UTC m=+80.773754723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0d04454e-e78a-4736-86fd-9cf2cc563221-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-96dmx" (UID: "0d04454e-e78a-4736-86fd-9cf2cc563221") : secret "monitoring-plugin-cert" not found
Sep 10 17:13:15.062106 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.062076    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lhz\" (UniqueName: \"kubernetes.io/projected/0d04454e-e78a-4736-86fd-9cf2cc563221-kube-api-access-j5lhz\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:15.557320 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.557276    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d04454e-e78a-4736-86fd-9cf2cc563221-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:15.559807 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.559782    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d04454e-e78a-4736-86fd-9cf2cc563221-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-96dmx\" (UID: \"0d04454e-e78a-4736-86fd-9cf2cc563221\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:15.722089 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.722046    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"32c699866dce4240ba81ae5ecf56a9a29e45e01a8ff41f788f8097c7a8763734"}
Sep 10 17:13:15.722535 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.722140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"817a2cffb6fc3677e33f6ac5dd0ff45dcc828efdd7da53ee980d2e0ba2897455"}
Sep 10 17:13:15.722535 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.722199    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"7bf122c35d9410bc04b22bd2ef806e306ba7955eed28b80642b63e6e2547356a"}
Sep 10 17:13:15.824723 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:15.824620    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:16.101196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.101169    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"]
Sep 10 17:13:16.361231 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:16.361157    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d04454e_e78a_4736_86fd_9cf2cc563221.slice/crio-d437f6f6369ea89b4a949d5482d6195a1edc653cc7160cb8f3e626ac9480ed7d WatchSource:0}: Error finding container d437f6f6369ea89b4a949d5482d6195a1edc653cc7160cb8f3e626ac9480ed7d: Status 404 returned error can't find the container with id d437f6f6369ea89b4a949d5482d6195a1edc653cc7160cb8f3e626ac9480ed7d
Sep 10 17:13:16.425326 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.425268    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:13:16.429132 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.429110    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.439369 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439339    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 17:13:16.439479 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439375    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 17:13:16.439479 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439438    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 17:13:16.439589 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439511    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 17:13:16.439589 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439546    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 17:13:16.439702 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439613    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 17:13:16.439702 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439552    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 17:13:16.439810 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439773    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8vfr1092aieg1\""
Sep 10 17:13:16.439972 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.439955    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 17:13:16.440070 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440056    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 17:13:16.440750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440553    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 17:13:16.440750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440554    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vv4mb\""
Sep 10 17:13:16.440750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440602    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 17:13:16.440750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 17:13:16.440750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440557    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 17:13:16.441089 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.440789    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 17:13:16.442829 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.442803    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 17:13:16.446764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.446741    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.465832    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-web-config\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.465877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.465911    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.465935    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.465965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.465997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466021    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466069    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466116    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466151    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466182    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466276    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-config\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466299    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blxgj\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-kube-api-access-blxgj\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466543    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-config-out\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.466873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.466566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.567923    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.567981    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568007    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568058    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568105    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568134    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-config\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568211    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blxgj\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-kube-api-access-blxgj\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568299    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568338    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-config-out\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568395    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-web-config\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568457    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.568737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.568515    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.571642 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.569076    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.571642 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.569560    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.572097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.571983    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.572850 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.572758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.573239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.573209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.573588 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.573327    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.573964 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.573923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.574381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.574354    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.575137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.575116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-web-config\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.576153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.575742    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.576153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.575838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-config\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.576153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.576115    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.576153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.576124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.576403 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.576383    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.576910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.576884    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.578102 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.577965    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.578263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.578231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-config-out\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.578993 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.578966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.582266 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.582246    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blxgj\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-kube-api-access-blxgj\") pod \"prometheus-k8s-0\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.729121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.729091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"}
Sep 10 17:13:16.729520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.729198    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"}
Sep 10 17:13:16.729520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.729227    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"}
Sep 10 17:13:16.729520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.729265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"}
Sep 10 17:13:16.729520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.729309    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"}
Sep 10 17:13:16.729520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.729335    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerStarted","Data":"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"}
Sep 10 17:13:16.732092 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.732067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"5bf4b4e330fa8c44d2941880cd8e2fdfca828806c28f724e69a9295e24140ad0"}
Sep 10 17:13:16.732175 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.732136    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"d379691152528ac207b0f1d3f5284c0d0c01e77981ffdc0cdd3a02392d5bfb05"}
Sep 10 17:13:16.732214 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.732176    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:16.732214 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.732207    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" event={"ID":"d6db4dea-e1d1-425c-9486-50297d78e145","Type":"ContainerStarted","Data":"a98b1c1e22f9093a30d8fa95ad95292f483b557f38b13e2009dadc6b7b433633"}
Sep 10 17:13:16.733248 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.733225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx" event={"ID":"0d04454e-e78a-4736-86fd-9cf2cc563221","Type":"ContainerStarted","Data":"d437f6f6369ea89b4a949d5482d6195a1edc653cc7160cb8f3e626ac9480ed7d"}
Sep 10 17:13:16.737911 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.737896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:16.758846 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.758783    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.872836509 podStartE2EDuration="5.7587639s" totalImagesPullingTime="2.865656648s" totalInitContainerRuntime="20.270743ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:16.758741606 +0000 UTC m=+81.978865544" watchObservedRunningTime="2026-09-10 17:13:16.7587639 +0000 UTC m=+81.978887836"
Sep 10 17:13:16.809857 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:16.809605    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr" podStartSLOduration=1.468307287 podStartE2EDuration="4.809583475s" totalImagesPullingTime="3.341276188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:16.80916459 +0000 UTC m=+82.029288546" watchObservedRunningTime="2026-09-10 17:13:16.809583475 +0000 UTC m=+82.029707409"
Sep 10 17:13:17.036133 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:17.036075    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:13:17.039823 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:13:17.039782    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd719ab40_e7fb_4348_9d62_639fb332a56f.slice/crio-66eaa68ce8ef2525ac6f75897cb289725dea218014f5ea1d5d6416370449be12 WatchSource:0}: Error finding container 66eaa68ce8ef2525ac6f75897cb289725dea218014f5ea1d5d6416370449be12: Status 404 returned error can't find the container with id 66eaa68ce8ef2525ac6f75897cb289725dea218014f5ea1d5d6416370449be12
Sep 10 17:13:17.616104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:17.616073    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5987474868-bvs76"
Sep 10 17:13:17.739778 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:17.739744    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="1e2d00e6721623a9810b2956b3fcc9c5b1468bc66e7a3e34cb4b33e8ba797ec5" exitCode=0
Sep 10 17:13:17.740214 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:17.739843    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"1e2d00e6721623a9810b2956b3fcc9c5b1468bc66e7a3e34cb4b33e8ba797ec5"}
Sep 10 17:13:17.740214 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:17.739958    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"66eaa68ce8ef2525ac6f75897cb289725dea218014f5ea1d5d6416370449be12"}
Sep 10 17:13:18.746281 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:18.746239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx" event={"ID":"0d04454e-e78a-4736-86fd-9cf2cc563221","Type":"ContainerStarted","Data":"00cb2095fc15d69912e81ec26d93fffe9e6a7b3983fc92919b62dd624c4e6ca4"}
Sep 10 17:13:18.746860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:18.746488    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:18.752036 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:18.752010    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx"
Sep 10 17:13:18.768845 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:18.768787    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-96dmx" podStartSLOduration=3.129627953 podStartE2EDuration="4.768771384s" totalImagesPullingTime="1.639143431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:18.767653116 +0000 UTC m=+83.987777053" watchObservedRunningTime="2026-09-10 17:13:18.768771384 +0000 UTC m=+83.988895320"
Sep 10 17:13:21.761719 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.761685    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"e72d00a2c6832ee0f9413abbdb16326d70dae595ac693cc4a52051ff96d9f794"}
Sep 10 17:13:21.762080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.761750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"161f527cddb4ab133317fe5692aa8a5d73d57f16af2a0b7a62e6b46974b49c0b"}
Sep 10 17:13:21.762080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.761781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"add931b04314e5fa7230507baaf030e5a41bab0d4c4fac6adec916818a79bc07"}
Sep 10 17:13:21.762080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.761805    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"0626910e74a5e68423b65c92453d3120b3cc6db960d68886d28f0bd0ba433b7f"}
Sep 10 17:13:21.762080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.761830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"32f87c644848232e815da970d7df5376fad729f7c06c25046743cf430f4ef5fe"}
Sep 10 17:13:21.762080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.761854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerStarted","Data":"b8f483c1610f985e65f9f5c976358365d9064d1a814c69f014a71e2803cbb1c0"}
Sep 10 17:13:21.794611 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:21.794560    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.507182975 podStartE2EDuration="5.794544424s" totalImagesPullingTime="3.269934388s" totalInitContainerRuntime="17.427061ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:13:21.793196241 +0000 UTC m=+87.013322161" watchObservedRunningTime="2026-09-10 17:13:21.794544424 +0000 UTC m=+87.014668357"
Sep 10 17:13:22.745846 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:22.745807    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b49f6574f-kxbmr"
Sep 10 17:13:26.738924 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:26.738878    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:13:38.675300 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:13:38.675265    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wxqdz"
Sep 10 17:14:16.738397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:16.738351    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:16.757956 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:16.757925    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:16.994150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:16.994082    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:30.497499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497465    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:14:30.498008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497869    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-metric" containerID="cri-o://4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6" gracePeriod=120
Sep 10 17:14:30.498008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497906    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-web" containerID="cri-o://c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb" gracePeriod=120
Sep 10 17:14:30.498008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497923    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy" containerID="cri-o://c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e" gracePeriod=120
Sep 10 17:14:30.498008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497937    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="config-reloader" containerID="cri-o://53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205" gracePeriod=120
Sep 10 17:14:30.498008 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497834    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="alertmanager" containerID="cri-o://2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a" gracePeriod=120
Sep 10 17:14:30.498252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:30.497995    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="prom-label-proxy" containerID="cri-o://2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288" gracePeriod=120
Sep 10 17:14:31.038985 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.038953    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288" exitCode=0
Sep 10 17:14:31.038985 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.038977    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e" exitCode=0
Sep 10 17:14:31.038985 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.038984    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205" exitCode=0
Sep 10 17:14:31.038985 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.038989    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a" exitCode=0
Sep 10 17:14:31.039234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.039024    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"}
Sep 10 17:14:31.039234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.039089    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"}
Sep 10 17:14:31.039234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.039117    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"}
Sep 10 17:14:31.039234 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.039141    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"}
Sep 10 17:14:31.735799 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.735777    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:31.889738 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889631    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-web\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.889884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889754    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-config-out\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.889884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889788    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-config-volume\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.889884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889813    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-tls-assets\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.889884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889845    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-trusted-ca-bundle\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.889884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889875    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-main-db\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889924    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889952    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-metric\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.889980    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g972f\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-kube-api-access-g972f\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890025    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-web-config\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890068    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-cluster-tls-config\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890097    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-metrics-client-ca\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890402 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890124    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls\") pod \"8343e387-06c0-4329-92e4-bf5dff4e4192\" (UID: \"8343e387-06c0-4329-92e4-bf5dff4e4192\") "
Sep 10 17:14:31.890402 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890255    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:14:31.890402 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890305    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:31.890564 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890455    2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.890564 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.890476    2556 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-alertmanager-main-db\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.891792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.891754    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:31.892901 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.892810    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.892901 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.892854    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.893100 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.892909    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:14:31.893260 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.893226    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.893456 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.893419    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-config-volume" (OuterVolumeSpecName: "config-volume") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.893593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.893576    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-config-out" (OuterVolumeSpecName: "config-out") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:14:31.894100 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.894073    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-kube-api-access-g972f" (OuterVolumeSpecName: "kube-api-access-g972f") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "kube-api-access-g972f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:14:31.894199 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.894174    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.897935 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.897859    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.903973 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.903952    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-web-config" (OuterVolumeSpecName: "web-config") pod "8343e387-06c0-4329-92e4-bf5dff4e4192" (UID: "8343e387-06c0-4329-92e4-bf5dff4e4192"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:31.991566 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991527    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991566 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991561    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991575    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g972f\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-kube-api-access-g972f\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991589    2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-web-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991605    2556 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-cluster-tls-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991616    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8343e387-06c0-4329-92e4-bf5dff4e4192-metrics-client-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991628    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-main-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991639    2556 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991650    2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8343e387-06c0-4329-92e4-bf5dff4e4192-config-out\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991677    2556 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8343e387-06c0-4329-92e4-bf5dff4e4192-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:31.991789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:31.991690    2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8343e387-06c0-4329-92e4-bf5dff4e4192-tls-assets\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:32.046299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046260    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6" exitCode=0
Sep 10 17:14:32.046299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046289    2556 generic.go:356] "Generic (PLEG): container finished" podID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerID="c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb" exitCode=0
Sep 10 17:14:32.046528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046340    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"}
Sep 10 17:14:32.046528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046406    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"}
Sep 10 17:14:32.046528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046348    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.046528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046444    2556 scope.go:117] "RemoveContainer" containerID="2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"
Sep 10 17:14:32.046528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.046435    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8343e387-06c0-4329-92e4-bf5dff4e4192","Type":"ContainerDied","Data":"713384f066057b9e6a6afda398e301b81c7efc112bab5f19ed71d06f880812cd"}
Sep 10 17:14:32.055037 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.055016    2556 scope.go:117] "RemoveContainer" containerID="4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"
Sep 10 17:14:32.061929 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.061912    2556 scope.go:117] "RemoveContainer" containerID="c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"
Sep 10 17:14:32.068866 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.068843    2556 scope.go:117] "RemoveContainer" containerID="c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"
Sep 10 17:14:32.070197 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.070142    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:14:32.075028 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.075002    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:14:32.077192 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.077170    2556 scope.go:117] "RemoveContainer" containerID="53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"
Sep 10 17:14:32.084256 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.084240    2556 scope.go:117] "RemoveContainer" containerID="2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"
Sep 10 17:14:32.091058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.091044    2556 scope.go:117] "RemoveContainer" containerID="323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"
Sep 10 17:14:32.098015 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.097993    2556 scope.go:117] "RemoveContainer" containerID="2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"
Sep 10 17:14:32.098277 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.098257    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288\": container with ID starting with 2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288 not found: ID does not exist" containerID="2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"
Sep 10 17:14:32.098356 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.098289    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"} err="failed to get container status \"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288\": rpc error: code = NotFound desc = could not find container \"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288\": container with ID starting with 2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288 not found: ID does not exist"
Sep 10 17:14:32.098356 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.098334    2556 scope.go:117] "RemoveContainer" containerID="4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"
Sep 10 17:14:32.098594 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.098577    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6\": container with ID starting with 4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6 not found: ID does not exist" containerID="4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"
Sep 10 17:14:32.098633 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.098600    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"} err="failed to get container status \"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6\": rpc error: code = NotFound desc = could not find container \"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6\": container with ID starting with 4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6 not found: ID does not exist"
Sep 10 17:14:32.098633 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.098616    2556 scope.go:117] "RemoveContainer" containerID="c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"
Sep 10 17:14:32.099010 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.098973    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e\": container with ID starting with c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e not found: ID does not exist" containerID="c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"
Sep 10 17:14:32.099081 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.099017    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"} err="failed to get container status \"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e\": rpc error: code = NotFound desc = could not find container \"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e\": container with ID starting with c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e not found: ID does not exist"
Sep 10 17:14:32.099081 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.099040    2556 scope.go:117] "RemoveContainer" containerID="c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"
Sep 10 17:14:32.099443 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.099411    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb\": container with ID starting with c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb not found: ID does not exist" containerID="c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"
Sep 10 17:14:32.099548 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.099442    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"} err="failed to get container status \"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb\": rpc error: code = NotFound desc = could not find container \"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb\": container with ID starting with c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb not found: ID does not exist"
Sep 10 17:14:32.099548 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.099464    2556 scope.go:117] "RemoveContainer" containerID="53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"
Sep 10 17:14:32.099809 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.099787    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205\": container with ID starting with 53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205 not found: ID does not exist" containerID="53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"
Sep 10 17:14:32.099961 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.099817    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"} err="failed to get container status \"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205\": rpc error: code = NotFound desc = could not find container \"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205\": container with ID starting with 53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205 not found: ID does not exist"
Sep 10 17:14:32.099961 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.099837    2556 scope.go:117] "RemoveContainer" containerID="2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"
Sep 10 17:14:32.100164 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.100143    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a\": container with ID starting with 2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a not found: ID does not exist" containerID="2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"
Sep 10 17:14:32.100242 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.100171    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"} err="failed to get container status \"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a\": rpc error: code = NotFound desc = could not find container \"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a\": container with ID starting with 2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a not found: ID does not exist"
Sep 10 17:14:32.100242 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.100190    2556 scope.go:117] "RemoveContainer" containerID="323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"
Sep 10 17:14:32.100467 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:14:32.100451    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a\": container with ID starting with 323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a not found: ID does not exist" containerID="323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"
Sep 10 17:14:32.100531 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.100473    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"} err="failed to get container status \"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a\": rpc error: code = NotFound desc = could not find container \"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a\": container with ID starting with 323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a not found: ID does not exist"
Sep 10 17:14:32.100531 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.100492    2556 scope.go:117] "RemoveContainer" containerID="2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"
Sep 10 17:14:32.100802 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.100782    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288"} err="failed to get container status \"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288\": rpc error: code = NotFound desc = could not find container \"2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288\": container with ID starting with 2c95e422f864fd7d29967b6757ccd1dee4f930cd66d31cae77050d13a4751288 not found: ID does not exist"
Sep 10 17:14:32.100894 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.100802    2556 scope.go:117] "RemoveContainer" containerID="4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"
Sep 10 17:14:32.101089 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101070    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6"} err="failed to get container status \"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6\": rpc error: code = NotFound desc = could not find container \"4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6\": container with ID starting with 4878e5e8debdd7b4cdc9b67a84fd7f2fa4f57f994708bd027fc36a9c97b923e6 not found: ID does not exist"
Sep 10 17:14:32.101141 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101090    2556 scope.go:117] "RemoveContainer" containerID="c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"
Sep 10 17:14:32.101185 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101168    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:14:32.101315 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101276    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e"} err="failed to get container status \"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e\": rpc error: code = NotFound desc = could not find container \"c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e\": container with ID starting with c327638e07bc71c5886ef33611287acbbf0d7b4b9fce6223e9f6be6887947c9e not found: ID does not exist"
Sep 10 17:14:32.101315 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101296    2556 scope.go:117] "RemoveContainer" containerID="c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"
Sep 10 17:14:32.101522 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101504    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb"} err="failed to get container status \"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb\": rpc error: code = NotFound desc = could not find container \"c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb\": container with ID starting with c7cf3fa1fe16d4f1e7228f5a1eb944df637014d9a6876daa084c1addd59a3abb not found: ID does not exist"
Sep 10 17:14:32.101590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101540    2556 scope.go:117] "RemoveContainer" containerID="53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"
Sep 10 17:14:32.101638 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101514    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy"
Sep 10 17:14:32.101638 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101602    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy"
Sep 10 17:14:32.101638 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101615    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="prom-label-proxy"
Sep 10 17:14:32.101638 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101624    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="prom-label-proxy"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101638    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-web"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101647    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-web"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101658    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="alertmanager"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101688    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="alertmanager"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101709    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="init-config-reloader"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101717    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="init-config-reloader"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101739    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-metric"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101747    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-metric"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101757    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="config-reloader"
Sep 10 17:14:32.101803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101765    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="config-reloader"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101832    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="alertmanager"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101849    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101856    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="prom-label-proxy"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101862    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="config-reloader"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101867    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-metric"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101873    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" containerName="kube-rbac-proxy-web"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101826    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205"} err="failed to get container status \"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205\": rpc error: code = NotFound desc = could not find container \"53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205\": container with ID starting with 53fde7c9799daf78dac3e92e447d3305a5105bb3a5bb0777845e1f2475d72205 not found: ID does not exist"
Sep 10 17:14:32.102139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.101916    2556 scope.go:117] "RemoveContainer" containerID="2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"
Sep 10 17:14:32.102441 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.102151    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a"} err="failed to get container status \"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a\": rpc error: code = NotFound desc = could not find container \"2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a\": container with ID starting with 2414cb16dd4501f827c920dcba543d4e82e2e22d658219f2cee8233beb683c0a not found: ID does not exist"
Sep 10 17:14:32.102441 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.102173    2556 scope.go:117] "RemoveContainer" containerID="323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"
Sep 10 17:14:32.102441 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.102389    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a"} err="failed to get container status \"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a\": rpc error: code = NotFound desc = could not find container \"323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a\": container with ID starting with 323f2e2ca2dd893827e109675919871d7d7c19ad3d26e18f7ba1eb1a207afb7a not found: ID does not exist"
Sep 10 17:14:32.105336 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.105322    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.107904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.107885    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 17:14:32.107997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.107889    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 17:14:32.108268 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108252    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 17:14:32.108412 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108397    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 17:14:32.108476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108427    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 17:14:32.108534 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108433    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 17:14:32.108534 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108504    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 17:14:32.108802 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108783    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-69fhq\""
Sep 10 17:14:32.108881 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.108787    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 17:14:32.113588 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.113566    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 17:14:32.122152 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.122128    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:14:32.192494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192402    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/417d8e16-cdbd-4d5d-a693-83d4988718a0-config-out\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192757 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192528    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192757 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192604    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/417d8e16-cdbd-4d5d-a693-83d4988718a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192757 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192757 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192686    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkz6\" (UniqueName: \"kubernetes.io/projected/417d8e16-cdbd-4d5d-a693-83d4988718a0-kube-api-access-stkz6\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192757 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192717    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/417d8e16-cdbd-4d5d-a693-83d4988718a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192810    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192835    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/417d8e16-cdbd-4d5d-a693-83d4988718a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417d8e16-cdbd-4d5d-a693-83d4988718a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.192982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.192979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-web-config\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293371 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-web-config\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293371 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/417d8e16-cdbd-4d5d-a693-83d4988718a0-config-out\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293426    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/417d8e16-cdbd-4d5d-a693-83d4988718a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkz6\" (UniqueName: \"kubernetes.io/projected/417d8e16-cdbd-4d5d-a693-83d4988718a0-kube-api-access-stkz6\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.293601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/417d8e16-cdbd-4d5d-a693-83d4988718a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.294346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293616    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.294346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293645    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.294346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/417d8e16-cdbd-4d5d-a693-83d4988718a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.294346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.293753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417d8e16-cdbd-4d5d-a693-83d4988718a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.294346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.294039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/417d8e16-cdbd-4d5d-a693-83d4988718a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.294595 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.294501    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/417d8e16-cdbd-4d5d-a693-83d4988718a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.295133 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.295035    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/417d8e16-cdbd-4d5d-a693-83d4988718a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.296705 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.296651    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/417d8e16-cdbd-4d5d-a693-83d4988718a0-config-out\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.296857 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.296795    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.297065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.297033    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.297169 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.297105    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-web-config\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.297169 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.297107    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.297324 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.297300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.297398 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.297329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/417d8e16-cdbd-4d5d-a693-83d4988718a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.297477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.297461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.298067 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.298050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/417d8e16-cdbd-4d5d-a693-83d4988718a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.304748 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.304732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkz6\" (UniqueName: \"kubernetes.io/projected/417d8e16-cdbd-4d5d-a693-83d4988718a0-kube-api-access-stkz6\") pod \"alertmanager-main-0\" (UID: \"417d8e16-cdbd-4d5d-a693-83d4988718a0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.414931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.414894    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 17:14:32.699092 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:32.699062    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 17:14:32.700019 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:14:32.699992    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d8e16_cdbd_4d5d_a693_83d4988718a0.slice/crio-a87303c88bd70beb17f05b924ed091409f405b266b605a85dbe3f9653965e539 WatchSource:0}: Error finding container a87303c88bd70beb17f05b924ed091409f405b266b605a85dbe3f9653965e539: Status 404 returned error can't find the container with id a87303c88bd70beb17f05b924ed091409f405b266b605a85dbe3f9653965e539
Sep 10 17:14:33.052657 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:33.052618    2556 generic.go:356] "Generic (PLEG): container finished" podID="417d8e16-cdbd-4d5d-a693-83d4988718a0" containerID="34ca2e1cfebe1d16992a4a1e49ba2697a0597eb99059a1cbd6e51c5ee1926c90" exitCode=0
Sep 10 17:14:33.053086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:33.052694    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerDied","Data":"34ca2e1cfebe1d16992a4a1e49ba2697a0597eb99059a1cbd6e51c5ee1926c90"}
Sep 10 17:14:33.053086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:33.052744    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"a87303c88bd70beb17f05b924ed091409f405b266b605a85dbe3f9653965e539"}
Sep 10 17:14:33.271745 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:33.271690    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8343e387-06c0-4329-92e4-bf5dff4e4192" path="/var/lib/kubelet/pods/8343e387-06c0-4329-92e4-bf5dff4e4192/volumes"
Sep 10 17:14:34.059853 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.059817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"7df0a4ef00059ca9ccf9074696b3305156d7f49f56fbc90589dd5deca20ec345"}
Sep 10 17:14:34.060239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.059882    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"8377a2702ac56f8a17f3c3c78c4782d8943e80386680fb9deca5db3059b879c8"}
Sep 10 17:14:34.060239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.059915    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"e9203d1c5488622a15b6cfa5b0e69d252cf0f55fceadf0b0a02a0659728448c0"}
Sep 10 17:14:34.060239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.059941    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"7e9707b3a8a6c6c4205ac2dcf387a96f18827a2a315f73f0e3a8eb4609259896"}
Sep 10 17:14:34.060239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.059964    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"12d4ceac3200e5f072bf4503d855bdde312b6cb98392c18d79bba7ba4bd7f5b9"}
Sep 10 17:14:34.060239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.059990    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"417d8e16-cdbd-4d5d-a693-83d4988718a0","Type":"ContainerStarted","Data":"955752aa2638f0a82fa49af6cfcd922f9a374038f5960dea1ff552bb76354a6c"}
Sep 10 17:14:34.088156 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.088099    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.069421449 podStartE2EDuration="2.088077851s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.656402ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:34.085324844 +0000 UTC m=+159.305448776" watchObservedRunningTime="2026-09-10 17:14:34.088077851 +0000 UTC m=+159.308201785"
Sep 10 17:14:34.525416 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.525373    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"]
Sep 10 17:14:34.527915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.527900    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.530256 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.530235    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 17:14:34.530381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.530242    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 17:14:34.530381 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.530304    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 17:14:34.530492 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.530376    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 17:14:34.530603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.530586    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9wdh7\""
Sep 10 17:14:34.530651 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.530638    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 17:14:34.535589 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.535567    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 17:14:34.543678 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.542552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"]
Sep 10 17:14:34.614032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.613960    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.613995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-secret-telemeter-client\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.614017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-metrics-client-ca\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.614068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-serving-certs-ca-bundle\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.614111    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-federate-client-tls\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.614132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-telemeter-client-tls\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.614148    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.614336 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.614221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dlk\" (UniqueName: \"kubernetes.io/projected/837c5b96-d480-4059-ac9f-88ec5262ebd7-kube-api-access-88dlk\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715325 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715288    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dlk\" (UniqueName: \"kubernetes.io/projected/837c5b96-d480-4059-ac9f-88ec5262ebd7-kube-api-access-88dlk\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715341    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715364    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-secret-telemeter-client\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-metrics-client-ca\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715407    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-serving-certs-ca-bundle\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-federate-client-tls\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715448    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-telemeter-client-tls\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.715480 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.715464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.716358 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.716323    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-metrics-client-ca\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.716358 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.716339    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-serving-certs-ca-bundle\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.716786 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.716760    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/837c5b96-d480-4059-ac9f-88ec5262ebd7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.718029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.718001    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-telemeter-client-tls\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.718117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.718074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-secret-telemeter-client\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.718117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.718103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-federate-client-tls\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.718213 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.718191    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/837c5b96-d480-4059-ac9f-88ec5262ebd7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.724158 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.724134    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dlk\" (UniqueName: \"kubernetes.io/projected/837c5b96-d480-4059-ac9f-88ec5262ebd7-kube-api-access-88dlk\") pod \"telemeter-client-78d6c7bd9d-s8bjd\" (UID: \"837c5b96-d480-4059-ac9f-88ec5262ebd7\") " pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.838200 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.838138    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"
Sep 10 17:14:34.841015 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.840987    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:14:34.841593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.841362    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-web" containerID="cri-o://add931b04314e5fa7230507baaf030e5a41bab0d4c4fac6adec916818a79bc07" gracePeriod=600
Sep 10 17:14:34.841593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.841372    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="config-reloader" containerID="cri-o://32f87c644848232e815da970d7df5376fad729f7c06c25046743cf430f4ef5fe" gracePeriod=600
Sep 10 17:14:34.841593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.841407    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy" containerID="cri-o://161f527cddb4ab133317fe5692aa8a5d73d57f16af2a0b7a62e6b46974b49c0b" gracePeriod=600
Sep 10 17:14:34.841593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.841385    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e72d00a2c6832ee0f9413abbdb16326d70dae595ac693cc4a52051ff96d9f794" gracePeriod=600
Sep 10 17:14:34.841593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.841362    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="thanos-sidecar" containerID="cri-o://0626910e74a5e68423b65c92453d3120b3cc6db960d68886d28f0bd0ba433b7f" gracePeriod=600
Sep 10 17:14:34.841593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:34.841359    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="prometheus" containerID="cri-o://b8f483c1610f985e65f9f5c976358365d9064d1a814c69f014a71e2803cbb1c0" gracePeriod=600
Sep 10 17:14:35.066928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066900    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="e72d00a2c6832ee0f9413abbdb16326d70dae595ac693cc4a52051ff96d9f794" exitCode=0
Sep 10 17:14:35.066928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066924    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="161f527cddb4ab133317fe5692aa8a5d73d57f16af2a0b7a62e6b46974b49c0b" exitCode=0
Sep 10 17:14:35.066928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066931    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="add931b04314e5fa7230507baaf030e5a41bab0d4c4fac6adec916818a79bc07" exitCode=0
Sep 10 17:14:35.066928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066936    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="0626910e74a5e68423b65c92453d3120b3cc6db960d68886d28f0bd0ba433b7f" exitCode=0
Sep 10 17:14:35.066928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066941    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="32f87c644848232e815da970d7df5376fad729f7c06c25046743cf430f4ef5fe" exitCode=0
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066946    2556 generic.go:356] "Generic (PLEG): container finished" podID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerID="b8f483c1610f985e65f9f5c976358365d9064d1a814c69f014a71e2803cbb1c0" exitCode=0
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.066975    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"e72d00a2c6832ee0f9413abbdb16326d70dae595ac693cc4a52051ff96d9f794"}
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.067067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"161f527cddb4ab133317fe5692aa8a5d73d57f16af2a0b7a62e6b46974b49c0b"}
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.067113    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"add931b04314e5fa7230507baaf030e5a41bab0d4c4fac6adec916818a79bc07"}
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.067156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"0626910e74a5e68423b65c92453d3120b3cc6db960d68886d28f0bd0ba433b7f"}
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.067215    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"32f87c644848232e815da970d7df5376fad729f7c06c25046743cf430f4ef5fe"}
Sep 10 17:14:35.067443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.067257    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"b8f483c1610f985e65f9f5c976358365d9064d1a814c69f014a71e2803cbb1c0"}
Sep 10 17:14:35.071500 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.071481    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:35.128704 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.128608    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd"]
Sep 10 17:14:35.131535 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:14:35.131508    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837c5b96_d480_4059_ac9f_88ec5262ebd7.slice/crio-97832da386830209a8bc2a4f975424f852510d0defcefa97db0cf66a464363be WatchSource:0}: Error finding container 97832da386830209a8bc2a4f975424f852510d0defcefa97db0cf66a464363be: Status 404 returned error can't find the container with id 97832da386830209a8bc2a4f975424f852510d0defcefa97db0cf66a464363be
Sep 10 17:14:35.219714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219688    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-thanos-prometheus-http-client-file\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.219886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219722    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-trusted-ca-bundle\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.219886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219763    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-config\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.219886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219780    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.219886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219834    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-tls\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.219886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219860    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-metrics-client-certs\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.219886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219883    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-config-out\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219915    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-metrics-client-ca\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219946    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-db\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.219976    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-tls-assets\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220006    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-1\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220050    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-0\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220082    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blxgj\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-kube-api-access-blxgj\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220109    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-2\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220139    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-web-config\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220161    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-kube-rbac-proxy\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-serving-certs-ca-bundle\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220215    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.220621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220225    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220287    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-grpc-tls\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220370    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-kubelet-serving-ca-bundle\") pod \"d719ab40-e7fb-4348-9d62-639fb332a56f\" (UID: \"d719ab40-e7fb-4348-9d62-639fb332a56f\") "
Sep 10 17:14:35.220621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220521    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.220986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220957    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.221043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.220997    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-metrics-client-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.221043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.221018    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.221902 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.221871    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.221991 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.221969    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:14:35.222195 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.222172    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.223332 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.223302    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.223442 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.223407    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:14:35.223622 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.223587    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-config" (OuterVolumeSpecName: "config") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.223770 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.223616    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.223849 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.223821    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-config-out" (OuterVolumeSpecName: "config-out") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:14:35.224280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.224202    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.224280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.224227    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-kube-api-access-blxgj" (OuterVolumeSpecName: "kube-api-access-blxgj") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "kube-api-access-blxgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:14:35.224280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.224240    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.224280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.224264    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.224583 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.224334    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:14:35.225036 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.225019    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.225124 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.225044    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.225624 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.225593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.235130 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.235112    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-web-config" (OuterVolumeSpecName: "web-config") pod "d719ab40-e7fb-4348-9d62-639fb332a56f" (UID: "d719ab40-e7fb-4348-9d62-639fb332a56f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:14:35.321538 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321515    2556 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321538 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321536    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321548    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321558    2556 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-metrics-client-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321567    2556 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-config-out\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321574    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-db\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321582    2556 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-tls-assets\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321590    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321598    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321606    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blxgj\" (UniqueName: \"kubernetes.io/projected/d719ab40-e7fb-4348-9d62-639fb332a56f-kube-api-access-blxgj\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321614    2556 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321622    2556 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-web-config\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321630    2556 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-kube-rbac-proxy\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321638    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321647    2556 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321656    2556 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-secret-grpc-tls\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321686    2556 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d719ab40-e7fb-4348-9d62-639fb332a56f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:35.321987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:35.321698    2556 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d719ab40-e7fb-4348-9d62-639fb332a56f-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:14:36.074908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.074828    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.074908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.074841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d719ab40-e7fb-4348-9d62-639fb332a56f","Type":"ContainerDied","Data":"66eaa68ce8ef2525ac6f75897cb289725dea218014f5ea1d5d6416370449be12"}
Sep 10 17:14:36.075390 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.074919    2556 scope.go:117] "RemoveContainer" containerID="e72d00a2c6832ee0f9413abbdb16326d70dae595ac693cc4a52051ff96d9f794"
Sep 10 17:14:36.076399 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.076224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd" event={"ID":"837c5b96-d480-4059-ac9f-88ec5262ebd7","Type":"ContainerStarted","Data":"97832da386830209a8bc2a4f975424f852510d0defcefa97db0cf66a464363be"}
Sep 10 17:14:36.082874 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.082847    2556 scope.go:117] "RemoveContainer" containerID="161f527cddb4ab133317fe5692aa8a5d73d57f16af2a0b7a62e6b46974b49c0b"
Sep 10 17:14:36.089699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.089683    2556 scope.go:117] "RemoveContainer" containerID="add931b04314e5fa7230507baaf030e5a41bab0d4c4fac6adec916818a79bc07"
Sep 10 17:14:36.096077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.096060    2556 scope.go:117] "RemoveContainer" containerID="0626910e74a5e68423b65c92453d3120b3cc6db960d68886d28f0bd0ba433b7f"
Sep 10 17:14:36.098284 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.098254    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:14:36.103172 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.103156    2556 scope.go:117] "RemoveContainer" containerID="32f87c644848232e815da970d7df5376fad729f7c06c25046743cf430f4ef5fe"
Sep 10 17:14:36.104973 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.104952    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:14:36.109649 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.109634    2556 scope.go:117] "RemoveContainer" containerID="b8f483c1610f985e65f9f5c976358365d9064d1a814c69f014a71e2803cbb1c0"
Sep 10 17:14:36.116441 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.116420    2556 scope.go:117] "RemoveContainer" containerID="1e2d00e6721623a9810b2956b3fcc9c5b1468bc66e7a3e34cb4b33e8ba797ec5"
Sep 10 17:14:36.143675 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143640    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:14:36.143950 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143937    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-web"
Sep 10 17:14:36.143990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143951    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-web"
Sep 10 17:14:36.143990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143960    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy"
Sep 10 17:14:36.143990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143966    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy"
Sep 10 17:14:36.143990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143979    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="prometheus"
Sep 10 17:14:36.143990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143985    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="prometheus"
Sep 10 17:14:36.143990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143991    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="thanos-sidecar"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.143997    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="thanos-sidecar"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144001    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-thanos"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144007    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-thanos"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144015    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="config-reloader"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144020    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="config-reloader"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144036    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="init-config-reloader"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144040    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="init-config-reloader"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144084    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="thanos-sidecar"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144090    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="prometheus"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144099    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="config-reloader"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144104    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-web"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144109    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy"
Sep 10 17:14:36.144165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.144115    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" containerName="kube-rbac-proxy-thanos"
Sep 10 17:14:36.147343 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.147329    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.151926 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.151885    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 17:14:36.152251 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.152231    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 17:14:36.152330 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.152283    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 17:14:36.155709 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.152484    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 17:14:36.155709 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.153995    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 17:14:36.155709 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.154214    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 17:14:36.156013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.155779    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vv4mb\""
Sep 10 17:14:36.156013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.155888    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 17:14:36.156217 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.156121    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 17:14:36.157969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.156369    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8vfr1092aieg1\""
Sep 10 17:14:36.157969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.156379    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 17:14:36.157969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.156547    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 17:14:36.157969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.157003    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 17:14:36.157969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.157176    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 17:14:36.160779 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.160752    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 17:14:36.164108 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.164088    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 17:14:36.166206 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.166158    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 17:14:36.183836 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.183804    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:14:36.331889 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.331889 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.331889 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j45f6\" (UniqueName: \"kubernetes.io/projected/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-kube-api-access-j45f6\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331898    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.331985    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332030    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332063    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-config\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332170    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332195    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332263    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332293    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.332382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.332324    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.432990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.432961    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.432998    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433015    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433104    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-config\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433509 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433509 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433714 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433693    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433756    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433780    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433828    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433857    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433879    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j45f6\" (UniqueName: \"kubernetes.io/projected/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-kube-api-access-j45f6\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433897    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.433923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433912    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.434398 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.433954    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.434398 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.434052    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.434847 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.434510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.434847 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.434556    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.435406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.435117    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.436883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.436531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.436883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.436704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.437465 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.437195    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.437894 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.437579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.437993 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.437895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.438321 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.438300    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.438642 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.438618    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.439063 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.439041    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-web-config\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.439323 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.439280    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.439768 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.439747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-config\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.439871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.439745    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-config-out\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.440159 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.440137    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.440495 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.440471    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.442868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.442843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j45f6\" (UniqueName: \"kubernetes.io/projected/40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6-kube-api-access-j45f6\") pod \"prometheus-k8s-0\" (UID: \"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.459812 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.459788    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:14:36.748385 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:36.748362    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 17:14:36.749945 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:14:36.749913    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40433bd3_fc5e_4d50_ba4e_f4f2a628e8e6.slice/crio-183fa5f43051f64d778a2095b4ed766fa7c6403352bf825d604aa18b91303585 WatchSource:0}: Error finding container 183fa5f43051f64d778a2095b4ed766fa7c6403352bf825d604aa18b91303585: Status 404 returned error can't find the container with id 183fa5f43051f64d778a2095b4ed766fa7c6403352bf825d604aa18b91303585
Sep 10 17:14:37.082406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:37.082321    2556 generic.go:356] "Generic (PLEG): container finished" podID="40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6" containerID="e686fff3a5245502aae4d7db2654aff8db85d2757d9661496c37965fbaef2a34" exitCode=0
Sep 10 17:14:37.082847 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:37.082418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerDied","Data":"e686fff3a5245502aae4d7db2654aff8db85d2757d9661496c37965fbaef2a34"}
Sep 10 17:14:37.082847 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:37.082526    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"183fa5f43051f64d778a2095b4ed766fa7c6403352bf825d604aa18b91303585"}
Sep 10 17:14:37.271472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:37.271439    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d719ab40-e7fb-4348-9d62-639fb332a56f" path="/var/lib/kubelet/pods/d719ab40-e7fb-4348-9d62-639fb332a56f/volumes"
Sep 10 17:14:38.095980 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.095940    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd" event={"ID":"837c5b96-d480-4059-ac9f-88ec5262ebd7","Type":"ContainerStarted","Data":"d33732127fbd762d6ef477d50eea02a78d4897741f7e74d1d3f09363fc22f9ea"}
Sep 10 17:14:38.096455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.096036    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd" event={"ID":"837c5b96-d480-4059-ac9f-88ec5262ebd7","Type":"ContainerStarted","Data":"8b7da2fb732653db5fc91118ce6428b7eb5911d84a6b62539201be427cd33d19"}
Sep 10 17:14:38.096455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.096067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd" event={"ID":"837c5b96-d480-4059-ac9f-88ec5262ebd7","Type":"ContainerStarted","Data":"45351a1fbc36609a8049198d444073b7548eca22b09c73bba137b7d32b667ae9"}
Sep 10 17:14:38.099471 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.099442    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"0f00b0dd86292fa85e3acf1e3f449ffcedb3e62f76cabf439a49be831fdcb6b1"}
Sep 10 17:14:38.099579 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.099499    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"f381319edd7e38028d376ca38a2b185a91cb63006d6eaa9ce3ac0a996923b711"}
Sep 10 17:14:38.099579 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.099529    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"78c4553276be4c7d9151668ca1188d8afb7b8a5df2930f8fe3231df287a2b515"}
Sep 10 17:14:38.099579 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.099553    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"77871231c017e3aa4ae41f9b7b5b90cc1f0bce441ddeed0387269e4e77b1d744"}
Sep 10 17:14:38.099579 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.099575    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"25cafd1e7e0bf139470aee50b60e01a5d12b91d0a8c271d65b758827d68077c7"}
Sep 10 17:14:38.099743 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.099597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6","Type":"ContainerStarted","Data":"e8e85b9a27fde4e15fc8fd5fed1c5a6b2b034f39bf6c4833f4b463da77d36ae8"}
Sep 10 17:14:38.118849 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.118802    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78d6c7bd9d-s8bjd" podStartSLOduration=1.7699174819999999 podStartE2EDuration="4.118787917s" totalImagesPullingTime="2.348870435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:38.117889511 +0000 UTC m=+163.338013443" watchObservedRunningTime="2026-09-10 17:14:38.118787917 +0000 UTC m=+163.338911849"
Sep 10 17:14:38.153814 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:38.153773    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.134951829 podStartE2EDuration="2.153758524s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.806695ms" isStatefulPod=false podCreationTimestamp="2026-09-10 17:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:14:38.150825894 +0000 UTC m=+163.370949827" watchObservedRunningTime="2026-09-10 17:14:38.153758524 +0000 UTC m=+163.373882457"
Sep 10 17:14:41.460633 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:14:41.460600    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:36.460133 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:15:36.460078    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:36.475486 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:15:36.475462    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:15:37.342290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:15:37.342263    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 17:16:55.168947 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:16:55.168912    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 17:17:42.076163 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.076125    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"]
Sep 10 17:17:42.079596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.079580    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.093324 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.093299    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tv4fm\""
Sep 10 17:17:42.093441 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.093361    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 17:17:42.093580 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.093565    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 17:17:42.093650 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.093608    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 17:17:42.093819 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.093805    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 17:17:42.152592 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.152567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"]
Sep 10 17:17:42.186658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.186631    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/884349a5-f6b9-47be-bc67-41aa70754038-tmp\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.186783 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.186687    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/884349a5-f6b9-47be-bc67-41aa70754038-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.186783 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.186717    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4d7x\" (UniqueName: \"kubernetes.io/projected/884349a5-f6b9-47be-bc67-41aa70754038-kube-api-access-f4d7x\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.186898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.186792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/884349a5-f6b9-47be-bc67-41aa70754038-manifests\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.186898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.186824    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/884349a5-f6b9-47be-bc67-41aa70754038-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.288013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.287985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/884349a5-f6b9-47be-bc67-41aa70754038-tmp\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.288117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.288017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/884349a5-f6b9-47be-bc67-41aa70754038-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.288117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.288035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4d7x\" (UniqueName: \"kubernetes.io/projected/884349a5-f6b9-47be-bc67-41aa70754038-kube-api-access-f4d7x\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.288117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.288066    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/884349a5-f6b9-47be-bc67-41aa70754038-manifests\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.288117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.288095    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/884349a5-f6b9-47be-bc67-41aa70754038-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.288468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.288448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/884349a5-f6b9-47be-bc67-41aa70754038-manifests\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.290378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.290356    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/884349a5-f6b9-47be-bc67-41aa70754038-tmp\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.290595 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.290574    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/884349a5-f6b9-47be-bc67-41aa70754038-apiservice-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.290681 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.290650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/884349a5-f6b9-47be-bc67-41aa70754038-webhook-cert\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.306916 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.306898    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4d7x\" (UniqueName: \"kubernetes.io/projected/884349a5-f6b9-47be-bc67-41aa70754038-kube-api-access-f4d7x\") pod \"opendatahub-operator-controller-manager-86f7548996-ftdv7\" (UID: \"884349a5-f6b9-47be-bc67-41aa70754038\") " pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.389067 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.388985    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:42.663707 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.663682    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"]
Sep 10 17:17:42.665892 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:17:42.665865    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884349a5_f6b9_47be_bc67_41aa70754038.slice/crio-5ee89581cb0eb190d9c04de79a7c4708a69af5dbf5ac23104520f20232d12ac2 WatchSource:0}: Error finding container 5ee89581cb0eb190d9c04de79a7c4708a69af5dbf5ac23104520f20232d12ac2: Status 404 returned error can't find the container with id 5ee89581cb0eb190d9c04de79a7c4708a69af5dbf5ac23104520f20232d12ac2
Sep 10 17:17:42.667594 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.667576    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:17:42.824366 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:42.824335    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7" event={"ID":"884349a5-f6b9-47be-bc67-41aa70754038","Type":"ContainerStarted","Data":"5ee89581cb0eb190d9c04de79a7c4708a69af5dbf5ac23104520f20232d12ac2"}
Sep 10 17:17:45.842346 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:45.842306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7" event={"ID":"884349a5-f6b9-47be-bc67-41aa70754038","Type":"ContainerStarted","Data":"13babe0cdd8d7779b3fc0440c7dcd3be551796f725ab2ac616ff053c518d30d6"}
Sep 10 17:17:46.848631 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:46.848593    2556 generic.go:356] "Generic (PLEG): container finished" podID="884349a5-f6b9-47be-bc67-41aa70754038" containerID="13babe0cdd8d7779b3fc0440c7dcd3be551796f725ab2ac616ff053c518d30d6" exitCode=0
Sep 10 17:17:46.849103 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:46.848693    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7" event={"ID":"884349a5-f6b9-47be-bc67-41aa70754038","Type":"ContainerDied","Data":"13babe0cdd8d7779b3fc0440c7dcd3be551796f725ab2ac616ff053c518d30d6"}
Sep 10 17:17:47.856547 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:47.856504    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7" event={"ID":"884349a5-f6b9-47be-bc67-41aa70754038","Type":"ContainerStarted","Data":"94d7a03543b5bec8d11ea8117088c04a3d21810cb27c23a08f98b41ae9fff3b4"}
Sep 10 17:17:47.857072 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:47.856596    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:17:47.892127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:47.892065    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7" podStartSLOduration=1.549821222 podStartE2EDuration="5.892048111s" totalImagesPullingTime="3.317095223s" totalInitContainerRuntime="1.025131666s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:17:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:17:47.890778436 +0000 UTC m=+353.110902368" watchObservedRunningTime="2026-09-10 17:17:47.892048111 +0000 UTC m=+353.112172043"
Sep 10 17:17:58.864051 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:17:58.864019    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-86f7548996-ftdv7"
Sep 10 17:18:00.787594 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.787563    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"]
Sep 10 17:18:00.796588 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.796564    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.800222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.800194    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 17:18:00.801089 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.801068    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 17:18:00.801189 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.801107    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 17:18:00.801469 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.801451    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-25vrv\""
Sep 10 17:18:00.801563 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.801503    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 17:18:00.801677 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.801626    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 17:18:00.803205 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.803176    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"]
Sep 10 17:18:00.849893 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.849861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmpf\" (UniqueName: \"kubernetes.io/projected/eca88365-fadb-443f-9f5a-663ddae8deae-kube-api-access-8nmpf\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.850065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.849905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eca88365-fadb-443f-9f5a-663ddae8deae-manager-config\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.850065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.849977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eca88365-fadb-443f-9f5a-663ddae8deae-metrics-cert\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.850065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.850019    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eca88365-fadb-443f-9f5a-663ddae8deae-cert\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.951049 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.951010    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eca88365-fadb-443f-9f5a-663ddae8deae-cert\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.951216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.951099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmpf\" (UniqueName: \"kubernetes.io/projected/eca88365-fadb-443f-9f5a-663ddae8deae-kube-api-access-8nmpf\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.951216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.951122    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eca88365-fadb-443f-9f5a-663ddae8deae-manager-config\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.951216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.951169    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eca88365-fadb-443f-9f5a-663ddae8deae-metrics-cert\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.952049 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.952017    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eca88365-fadb-443f-9f5a-663ddae8deae-manager-config\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.953652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.953633    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eca88365-fadb-443f-9f5a-663ddae8deae-metrics-cert\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.953767 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.953748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eca88365-fadb-443f-9f5a-663ddae8deae-cert\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:00.976804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:00.976779    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmpf\" (UniqueName: \"kubernetes.io/projected/eca88365-fadb-443f-9f5a-663ddae8deae-kube-api-access-8nmpf\") pod \"lws-controller-manager-9fc458d94-phwhm\" (UID: \"eca88365-fadb-443f-9f5a-663ddae8deae\") " pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:01.106190 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.106104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:01.402746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.402713    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"]
Sep 10 17:18:01.405470 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:18:01.405420    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca88365_fadb_443f_9f5a_663ddae8deae.slice/crio-8077db1b23ba28540799647157f4adca7f90c1444ef05a3eab8486ccbb48b1bd WatchSource:0}: Error finding container 8077db1b23ba28540799647157f4adca7f90c1444ef05a3eab8486ccbb48b1bd: Status 404 returned error can't find the container with id 8077db1b23ba28540799647157f4adca7f90c1444ef05a3eab8486ccbb48b1bd
Sep 10 17:18:01.728352 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.728317    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"]
Sep 10 17:18:01.732023 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.732004    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.737418 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.737395    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 17:18:01.737848 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.737828    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 17:18:01.737961 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.737923    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5tpbv\""
Sep 10 17:18:01.741820 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.741804    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 17:18:01.742588 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.742573    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 17:18:01.752763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.752742    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"]
Sep 10 17:18:01.756989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.756966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f4db00-e12f-4410-8ec1-4d334ad686a9-tls-certs\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.757077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.757011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06f4db00-e12f-4410-8ec1-4d334ad686a9-tmp\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.757145 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.757095    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtlj\" (UniqueName: \"kubernetes.io/projected/06f4db00-e12f-4410-8ec1-4d334ad686a9-kube-api-access-hgtlj\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.857695 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.857653    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06f4db00-e12f-4410-8ec1-4d334ad686a9-tmp\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.858117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.857719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtlj\" (UniqueName: \"kubernetes.io/projected/06f4db00-e12f-4410-8ec1-4d334ad686a9-kube-api-access-hgtlj\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.858117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.857792    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f4db00-e12f-4410-8ec1-4d334ad686a9-tls-certs\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.858117 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:18:01.857884    2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 17:18:01.858117 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:18:01.857945    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f4db00-e12f-4410-8ec1-4d334ad686a9-tls-certs podName:06f4db00-e12f-4410-8ec1-4d334ad686a9 nodeName:}" failed. No retries permitted until 2026-09-10 17:18:02.357926356 +0000 UTC m=+367.578050280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/06f4db00-e12f-4410-8ec1-4d334ad686a9-tls-certs") pod "kube-auth-proxy-647997b56d-8xz8b" (UID: "06f4db00-e12f-4410-8ec1-4d334ad686a9") : secret "kube-auth-proxy-tls" not found
Sep 10 17:18:01.859904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.859882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06f4db00-e12f-4410-8ec1-4d334ad686a9-tmp\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.877287 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.877259    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtlj\" (UniqueName: \"kubernetes.io/projected/06f4db00-e12f-4410-8ec1-4d334ad686a9-kube-api-access-hgtlj\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:01.923524 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:01.923497    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm" event={"ID":"eca88365-fadb-443f-9f5a-663ddae8deae","Type":"ContainerStarted","Data":"8077db1b23ba28540799647157f4adca7f90c1444ef05a3eab8486ccbb48b1bd"}
Sep 10 17:18:02.362105 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:02.362073    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f4db00-e12f-4410-8ec1-4d334ad686a9-tls-certs\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:02.364732 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:02.364704    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f4db00-e12f-4410-8ec1-4d334ad686a9-tls-certs\") pod \"kube-auth-proxy-647997b56d-8xz8b\" (UID: \"06f4db00-e12f-4410-8ec1-4d334ad686a9\") " pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:02.643002 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:02.642910    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"
Sep 10 17:18:02.948556 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:02.948522    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-647997b56d-8xz8b"]
Sep 10 17:18:02.950597 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:18:02.950567    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f4db00_e12f_4410_8ec1_4d334ad686a9.slice/crio-6395666dd0efb831918a05928c34cc81673dd26de9b9d4949f83b83ed61f35cf WatchSource:0}: Error finding container 6395666dd0efb831918a05928c34cc81673dd26de9b9d4949f83b83ed61f35cf: Status 404 returned error can't find the container with id 6395666dd0efb831918a05928c34cc81673dd26de9b9d4949f83b83ed61f35cf
Sep 10 17:18:03.934999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:03.934961    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b" event={"ID":"06f4db00-e12f-4410-8ec1-4d334ad686a9","Type":"ContainerStarted","Data":"6395666dd0efb831918a05928c34cc81673dd26de9b9d4949f83b83ed61f35cf"}
Sep 10 17:18:04.942698 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:04.942636    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm" event={"ID":"eca88365-fadb-443f-9f5a-663ddae8deae","Type":"ContainerStarted","Data":"6deff41a9b9ff97929c477ab7a17393c838afdc48a5a1e2e3dc17a5f941c2ee9"}
Sep 10 17:18:04.943201 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:04.942753    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:18:04.966821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:04.966767    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm" podStartSLOduration=2.279070999 podStartE2EDuration="4.966749093s" totalImagesPullingTime="2.687678094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:04.966707333 +0000 UTC m=+370.186831268" watchObservedRunningTime="2026-09-10 17:18:04.966749093 +0000 UTC m=+370.186873026"
Sep 10 17:18:06.953215 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:06.953179    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b" event={"ID":"06f4db00-e12f-4410-8ec1-4d334ad686a9","Type":"ContainerStarted","Data":"fea70c773e02170445126c9217353a26297e8d8120aa0533aab9524c6420e8a9"}
Sep 10 17:18:06.975305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:06.975251    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-647997b56d-8xz8b" podStartSLOduration=2.712235306 podStartE2EDuration="5.975235921s" totalImagesPullingTime="3.263000615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:18:06.974951082 +0000 UTC m=+372.195075040" watchObservedRunningTime="2026-09-10 17:18:06.975235921 +0000 UTC m=+372.195359853"
Sep 10 17:18:15.950247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:18:15.950216    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9fc458d94-phwhm"
Sep 10 17:19:55.663676 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.663626    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"]
Sep 10 17:19:55.666038 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.666019    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.669646 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.669625    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jfn2p\""
Sep 10 17:19:55.669879 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.669863    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 17:19:55.670218 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.670201    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 17:19:55.692091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.692070    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"]
Sep 10 17:19:55.760984 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.760956    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sntdr\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.761088 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.761017    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7j85\" (UniqueName: \"kubernetes.io/projected/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-kube-api-access-z7j85\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sntdr\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.862277 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.862245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7j85\" (UniqueName: \"kubernetes.io/projected/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-kube-api-access-z7j85\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sntdr\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.862411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.862324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sntdr\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.862703 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.862650    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sntdr\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.872060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.872043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7j85\" (UniqueName: \"kubernetes.io/projected/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-kube-api-access-z7j85\") pod \"kuadrant-operator-controller-manager-5769f99d9f-sntdr\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:55.975566 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:55.975541    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:19:56.257063 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:56.256910    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"]
Sep 10 17:19:56.259739 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:19:56.259705    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4831a7a_ae3b_42d1_9484_8752ed63bd1b.slice/crio-b4d99597bd209a94bbdf7f1973786cf8d0e0a9d91932fd5b364703884a56068c WatchSource:0}: Error finding container b4d99597bd209a94bbdf7f1973786cf8d0e0a9d91932fd5b364703884a56068c: Status 404 returned error can't find the container with id b4d99597bd209a94bbdf7f1973786cf8d0e0a9d91932fd5b364703884a56068c
Sep 10 17:19:56.447288 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:19:56.447254    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr" event={"ID":"a4831a7a-ae3b-42d1-9484-8752ed63bd1b","Type":"ContainerStarted","Data":"b4d99597bd209a94bbdf7f1973786cf8d0e0a9d91932fd5b364703884a56068c"}
Sep 10 17:20:00.474255 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:00.474219    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr" event={"ID":"a4831a7a-ae3b-42d1-9484-8752ed63bd1b","Type":"ContainerStarted","Data":"13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b"}
Sep 10 17:20:00.474597 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:00.474350    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:20:00.496761 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:00.496718    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr" podStartSLOduration=1.788217878 podStartE2EDuration="5.496705171s" totalImagesPullingTime="3.708487293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:00.494975148 +0000 UTC m=+485.715099081" watchObservedRunningTime="2026-09-10 17:20:00.496705171 +0000 UTC m=+485.716829104"
Sep 10 17:20:11.481168 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:11.481129    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:20:31.994467 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:31.994436    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-lqds6"]
Sep 10 17:20:31.998761 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:31.998740    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:32.001565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.001540    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vbl9d\""
Sep 10 17:20:32.022393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.022368    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lqds6"]
Sep 10 17:20:32.083524 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.083501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnx9c\" (UniqueName: \"kubernetes.io/projected/1d993ffc-89f3-411b-9b9e-e06159b26188-kube-api-access-hnx9c\") pod \"authorino-59c974bd48-lqds6\" (UID: \"1d993ffc-89f3-411b-9b9e-e06159b26188\") " pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:32.183862 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.183838    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnx9c\" (UniqueName: \"kubernetes.io/projected/1d993ffc-89f3-411b-9b9e-e06159b26188-kube-api-access-hnx9c\") pod \"authorino-59c974bd48-lqds6\" (UID: \"1d993ffc-89f3-411b-9b9e-e06159b26188\") " pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:32.195735 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.195708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnx9c\" (UniqueName: \"kubernetes.io/projected/1d993ffc-89f3-411b-9b9e-e06159b26188-kube-api-access-hnx9c\") pod \"authorino-59c974bd48-lqds6\" (UID: \"1d993ffc-89f3-411b-9b9e-e06159b26188\") " pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:32.307977 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.307915    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:32.597974 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.597906    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lqds6"]
Sep 10 17:20:32.603747 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:20:32.603719    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d993ffc_89f3_411b_9b9e_e06159b26188.slice/crio-f93ee32b9eee7453ca40dbb9200b4530349f7400374dd6701175cc4419f36bb8 WatchSource:0}: Error finding container f93ee32b9eee7453ca40dbb9200b4530349f7400374dd6701175cc4419f36bb8: Status 404 returned error can't find the container with id f93ee32b9eee7453ca40dbb9200b4530349f7400374dd6701175cc4419f36bb8
Sep 10 17:20:32.620758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:32.620728    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lqds6" event={"ID":"1d993ffc-89f3-411b-9b9e-e06159b26188","Type":"ContainerStarted","Data":"f93ee32b9eee7453ca40dbb9200b4530349f7400374dd6701175cc4419f36bb8"}
Sep 10 17:20:34.632641 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:34.632599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lqds6" event={"ID":"1d993ffc-89f3-411b-9b9e-e06159b26188","Type":"ContainerStarted","Data":"7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870"}
Sep 10 17:20:59.293445 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.293341    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-lqds6" podStartSLOduration=26.83133542 podStartE2EDuration="28.293324295s" totalImagesPullingTime="1.461988875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:20:34.664722457 +0000 UTC m=+519.884846387" watchObservedRunningTime="2026-09-10 17:20:59.293324295 +0000 UTC m=+544.513448227"
Sep 10 17:20:59.294249 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.294231    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lqds6"]
Sep 10 17:20:59.294416 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.294399    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-lqds6" podUID="1d993ffc-89f3-411b-9b9e-e06159b26188" containerName="authorino" containerID="cri-o://7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870" gracePeriod=30
Sep 10 17:20:59.534833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.534810    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:59.630565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.630472    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnx9c\" (UniqueName: \"kubernetes.io/projected/1d993ffc-89f3-411b-9b9e-e06159b26188-kube-api-access-hnx9c\") pod \"1d993ffc-89f3-411b-9b9e-e06159b26188\" (UID: \"1d993ffc-89f3-411b-9b9e-e06159b26188\") "
Sep 10 17:20:59.632676 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.632635    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d993ffc-89f3-411b-9b9e-e06159b26188-kube-api-access-hnx9c" (OuterVolumeSpecName: "kube-api-access-hnx9c") pod "1d993ffc-89f3-411b-9b9e-e06159b26188" (UID: "1d993ffc-89f3-411b-9b9e-e06159b26188"). InnerVolumeSpecName "kube-api-access-hnx9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:20:59.731286 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.731254    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnx9c\" (UniqueName: \"kubernetes.io/projected/1d993ffc-89f3-411b-9b9e-e06159b26188-kube-api-access-hnx9c\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:20:59.743163 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.743131    2556 generic.go:356] "Generic (PLEG): container finished" podID="1d993ffc-89f3-411b-9b9e-e06159b26188" containerID="7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870" exitCode=0
Sep 10 17:20:59.743299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.743167    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-lqds6"
Sep 10 17:20:59.743299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.743212    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lqds6" event={"ID":"1d993ffc-89f3-411b-9b9e-e06159b26188","Type":"ContainerDied","Data":"7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870"}
Sep 10 17:20:59.743299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.743252    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-lqds6" event={"ID":"1d993ffc-89f3-411b-9b9e-e06159b26188","Type":"ContainerDied","Data":"f93ee32b9eee7453ca40dbb9200b4530349f7400374dd6701175cc4419f36bb8"}
Sep 10 17:20:59.743299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.743272    2556 scope.go:117] "RemoveContainer" containerID="7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870"
Sep 10 17:20:59.752195 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.752178    2556 scope.go:117] "RemoveContainer" containerID="7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870"
Sep 10 17:20:59.752480 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:20:59.752462    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870\": container with ID starting with 7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870 not found: ID does not exist" containerID="7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870"
Sep 10 17:20:59.752528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.752491    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870"} err="failed to get container status \"7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870\": rpc error: code = NotFound desc = could not find container \"7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870\": container with ID starting with 7848186a09791881f2b2d0769f27064464d2078e6811bae1fa957b9db5c01870 not found: ID does not exist"
Sep 10 17:20:59.772514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.772488    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lqds6"]
Sep 10 17:20:59.778907 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:20:59.778885    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-lqds6"]
Sep 10 17:21:01.270825 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:01.270789    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d993ffc-89f3-411b-9b9e-e06159b26188" path="/var/lib/kubelet/pods/1d993ffc-89f3-411b-9b9e-e06159b26188/volumes"
Sep 10 17:21:44.387703 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.387651    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"]
Sep 10 17:21:44.388121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.388025    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d993ffc-89f3-411b-9b9e-e06159b26188" containerName="authorino"
Sep 10 17:21:44.388121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.388036    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d993ffc-89f3-411b-9b9e-e06159b26188" containerName="authorino"
Sep 10 17:21:44.388121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.388090    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d993ffc-89f3-411b-9b9e-e06159b26188" containerName="authorino"
Sep 10 17:21:44.391306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.391285    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.401507 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.401483    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 17:21:44.401610 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.401514    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l6qqm\""
Sep 10 17:21:44.479643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.479607    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"]
Sep 10 17:21:44.503520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.503488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391069a5-647f-4d8d-8111-7d6caa56d2a2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4kjb5\" (UID: \"391069a5-647f-4d8d-8111-7d6caa56d2a2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.503520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.503528    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85txp\" (UniqueName: \"kubernetes.io/projected/391069a5-647f-4d8d-8111-7d6caa56d2a2-kube-api-access-85txp\") pod \"payload-pre-processing-59fbc47bf8-4kjb5\" (UID: \"391069a5-647f-4d8d-8111-7d6caa56d2a2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.570115 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.570082    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"]
Sep 10 17:21:44.573378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.573362    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.602073 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.602049    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"]
Sep 10 17:21:44.604601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.604580    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391069a5-647f-4d8d-8111-7d6caa56d2a2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4kjb5\" (UID: \"391069a5-647f-4d8d-8111-7d6caa56d2a2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.604723 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.604612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85txp\" (UniqueName: \"kubernetes.io/projected/391069a5-647f-4d8d-8111-7d6caa56d2a2-kube-api-access-85txp\") pod \"payload-pre-processing-59fbc47bf8-4kjb5\" (UID: \"391069a5-647f-4d8d-8111-7d6caa56d2a2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.605519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.605497    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/391069a5-647f-4d8d-8111-7d6caa56d2a2-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-4kjb5\" (UID: \"391069a5-647f-4d8d-8111-7d6caa56d2a2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.666977 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.666900    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85txp\" (UniqueName: \"kubernetes.io/projected/391069a5-647f-4d8d-8111-7d6caa56d2a2-kube-api-access-85txp\") pod \"payload-pre-processing-59fbc47bf8-4kjb5\" (UID: \"391069a5-647f-4d8d-8111-7d6caa56d2a2\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.700609 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.700578    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:44.705997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.705968    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97929e09-8ffb-4a86-b440-b2bed7e9a3ac-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wjvg7\" (UID: \"97929e09-8ffb-4a86-b440-b2bed7e9a3ac\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.706125 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.706009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hhfg\" (UniqueName: \"kubernetes.io/projected/97929e09-8ffb-4a86-b440-b2bed7e9a3ac-kube-api-access-8hhfg\") pod \"payload-processing-5b54b7fbb7-wjvg7\" (UID: \"97929e09-8ffb-4a86-b440-b2bed7e9a3ac\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.806750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.806717    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97929e09-8ffb-4a86-b440-b2bed7e9a3ac-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wjvg7\" (UID: \"97929e09-8ffb-4a86-b440-b2bed7e9a3ac\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.806750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.806753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hhfg\" (UniqueName: \"kubernetes.io/projected/97929e09-8ffb-4a86-b440-b2bed7e9a3ac-kube-api-access-8hhfg\") pod \"payload-processing-5b54b7fbb7-wjvg7\" (UID: \"97929e09-8ffb-4a86-b440-b2bed7e9a3ac\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.807481 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.807452    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97929e09-8ffb-4a86-b440-b2bed7e9a3ac-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wjvg7\" (UID: \"97929e09-8ffb-4a86-b440-b2bed7e9a3ac\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.831780 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.831744    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hhfg\" (UniqueName: \"kubernetes.io/projected/97929e09-8ffb-4a86-b440-b2bed7e9a3ac-kube-api-access-8hhfg\") pod \"payload-processing-5b54b7fbb7-wjvg7\" (UID: \"97929e09-8ffb-4a86-b440-b2bed7e9a3ac\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:44.883223 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:44.883191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:45.017614 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:45.017588    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"]
Sep 10 17:21:45.022271 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:21:45.022240    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391069a5_647f_4d8d_8111_7d6caa56d2a2.slice/crio-81c4edb92ef5569448b530ba9ea14a63a8aa6faea1eac315f36b3460700a4f3f WatchSource:0}: Error finding container 81c4edb92ef5569448b530ba9ea14a63a8aa6faea1eac315f36b3460700a4f3f: Status 404 returned error can't find the container with id 81c4edb92ef5569448b530ba9ea14a63a8aa6faea1eac315f36b3460700a4f3f
Sep 10 17:21:45.179263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:45.179240    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"]
Sep 10 17:21:45.181645 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:21:45.181580    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97929e09_8ffb_4a86_b440_b2bed7e9a3ac.slice/crio-7a17e603003bb0af3536ee35d72e1290c45067afdb99a1ee45700fe722439ab5 WatchSource:0}: Error finding container 7a17e603003bb0af3536ee35d72e1290c45067afdb99a1ee45700fe722439ab5: Status 404 returned error can't find the container with id 7a17e603003bb0af3536ee35d72e1290c45067afdb99a1ee45700fe722439ab5
Sep 10 17:21:45.959045 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:45.959000    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5" event={"ID":"391069a5-647f-4d8d-8111-7d6caa56d2a2","Type":"ContainerStarted","Data":"81c4edb92ef5569448b530ba9ea14a63a8aa6faea1eac315f36b3460700a4f3f"}
Sep 10 17:21:45.961896 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:45.961859    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7" event={"ID":"97929e09-8ffb-4a86-b440-b2bed7e9a3ac","Type":"ContainerStarted","Data":"7a17e603003bb0af3536ee35d72e1290c45067afdb99a1ee45700fe722439ab5"}
Sep 10 17:21:47.974186 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:47.974148    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7" event={"ID":"97929e09-8ffb-4a86-b440-b2bed7e9a3ac","Type":"ContainerStarted","Data":"46f56d1f855746f74834a77f58ec6875c99454d197444fba1b63994ab0793ead"}
Sep 10 17:21:47.974647 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:47.974219    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:47.975819 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:47.975794    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5" event={"ID":"391069a5-647f-4d8d-8111-7d6caa56d2a2","Type":"ContainerStarted","Data":"03fc9f8e49db8f1d6796a62b1bb94bdfd502b9d3db22ca8e8a60c8b140c9e8f0"}
Sep 10 17:21:47.975919 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:47.975898    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:21:48.044777 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:48.044718    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5" podStartSLOduration=1.416416588 podStartE2EDuration="4.044700942s" totalImagesPullingTime="2.628284354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:48.043900073 +0000 UTC m=+593.264024014" watchObservedRunningTime="2026-09-10 17:21:48.044700942 +0000 UTC m=+593.264824875"
Sep 10 17:21:48.044978 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:48.044802    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7" podStartSLOduration=1.569144857 podStartE2EDuration="4.044798073s" totalImagesPullingTime="2.475653216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:21:48.01239334 +0000 UTC m=+593.232517274" watchObservedRunningTime="2026-09-10 17:21:48.044798073 +0000 UTC m=+593.264922006"
Sep 10 17:21:58.983840 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:58.983803    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-wjvg7"
Sep 10 17:21:58.984224 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:21:58.983959    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-4kjb5"
Sep 10 17:22:25.499315 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.499280    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"]
Sep 10 17:22:25.502853 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.502836    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.511359 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.511339    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 17:22:25.512278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.512259    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mn569\""
Sep 10 17:22:25.512278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.512269    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 17:22:25.512448 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.512317    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 17:22:25.526873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.526850    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"]
Sep 10 17:22:25.653072 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.653042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.653226 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.653093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.653226 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.653128    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8gl\" (UniqueName: \"kubernetes.io/projected/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-kube-api-access-cx8gl\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.653226 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.653160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.653226 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.653174    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.653363 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.653222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.753957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.753989    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8gl\" (UniqueName: \"kubernetes.io/projected/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-kube-api-access-cx8gl\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754314 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754450 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754388    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754450 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754450 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.754701 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.754649    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.756528 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.756505    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.756783 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.756767    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.771455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.771435    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8gl\" (UniqueName: \"kubernetes.io/projected/5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb-kube-api-access-cx8gl\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq\" (UID: \"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:25.812382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:25.812358    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:26.098921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:26.098867    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"]
Sep 10 17:22:26.099455 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:22:26.099427    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bed72cb_5e79_4c62_ae6f_2ecfca10c4eb.slice/crio-88edecf6b0fa753a26cba32b34d3d1e9c6889222894fcdbec05958558f3ff7e1 WatchSource:0}: Error finding container 88edecf6b0fa753a26cba32b34d3d1e9c6889222894fcdbec05958558f3ff7e1: Status 404 returned error can't find the container with id 88edecf6b0fa753a26cba32b34d3d1e9c6889222894fcdbec05958558f3ff7e1
Sep 10 17:22:26.164992 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:26.164957    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq" event={"ID":"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb","Type":"ContainerStarted","Data":"88edecf6b0fa753a26cba32b34d3d1e9c6889222894fcdbec05958558f3ff7e1"}
Sep 10 17:22:28.176774 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:28.176736    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq" event={"ID":"5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb","Type":"ContainerStarted","Data":"31255fec0e8fc40c1969944dace81b39984c2d0986a52e844c3b5b88204b3a13"}
Sep 10 17:22:28.204203 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:28.204151    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq" podStartSLOduration=2.133031228 podStartE2EDuration="3.204136929s" totalImagesPullingTime="1.071105701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:22:28.20408055 +0000 UTC m=+633.424204482" watchObservedRunningTime="2026-09-10 17:22:28.204136929 +0000 UTC m=+633.424260862"
Sep 10 17:22:35.812944 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:35.812899    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:35.813428 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:35.813059    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:35.825081 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:35.825056    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:22:36.225913 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:22:36.225885    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq"
Sep 10 17:23:10.356867 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.356833    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"]
Sep 10 17:23:10.359380 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.359356    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.362485 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.362461    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 17:23:10.380719 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.380693    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"]
Sep 10 17:23:10.436974 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.436942    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0715c7-1ba9-4a17-b5be-e164f774f20e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.437149 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.436993    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.437149 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.437106    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.437245 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.437148    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.437245 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.437172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.437245 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.437215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhvsx\" (UniqueName: \"kubernetes.io/projected/ed0715c7-1ba9-4a17-b5be-e164f774f20e-kube-api-access-vhvsx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538215    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538279    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhvsx\" (UniqueName: \"kubernetes.io/projected/ed0715c7-1ba9-4a17-b5be-e164f774f20e-kube-api-access-vhvsx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0715c7-1ba9-4a17-b5be-e164f774f20e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538468    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538712 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538656    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538765 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538726    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.538854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.538829    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.540547 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.540528    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed0715c7-1ba9-4a17-b5be-e164f774f20e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.540873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.540856    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed0715c7-1ba9-4a17-b5be-e164f774f20e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.547998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.547975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhvsx\" (UniqueName: \"kubernetes.io/projected/ed0715c7-1ba9-4a17-b5be-e164f774f20e-kube-api-access-vhvsx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk\" (UID: \"ed0715c7-1ba9-4a17-b5be-e164f774f20e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.668587 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.668499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:10.947918 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.947890    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"]
Sep 10 17:23:10.950018 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:23:10.949988    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded0715c7_1ba9_4a17_b5be_e164f774f20e.slice/crio-2c8ce4f4a812db81d05c59b3ee59290f5e2e1c6a54790ad424e369bc423e160e WatchSource:0}: Error finding container 2c8ce4f4a812db81d05c59b3ee59290f5e2e1c6a54790ad424e369bc423e160e: Status 404 returned error can't find the container with id 2c8ce4f4a812db81d05c59b3ee59290f5e2e1c6a54790ad424e369bc423e160e
Sep 10 17:23:10.951403 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:10.951382    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:23:11.375993 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:11.375956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk" event={"ID":"ed0715c7-1ba9-4a17-b5be-e164f774f20e","Type":"ContainerStarted","Data":"320f888e05eec8b50821aec8e62e5bac413de31050cd52d787cd1019f0b1f593"}
Sep 10 17:23:11.375993 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:11.375997    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk" event={"ID":"ed0715c7-1ba9-4a17-b5be-e164f774f20e","Type":"ContainerStarted","Data":"2c8ce4f4a812db81d05c59b3ee59290f5e2e1c6a54790ad424e369bc423e160e"}
Sep 10 17:23:20.669223 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:20.669189    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:20.669741 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:20.669262    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:20.682149 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:20.682122    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:20.707561 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:20.707515    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk" podStartSLOduration=10.52836087 podStartE2EDuration="10.707501831s" totalImagesPullingTime="179.140961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:23:11.402086595 +0000 UTC m=+676.622210539" watchObservedRunningTime="2026-09-10 17:23:20.707501831 +0000 UTC m=+685.927625764"
Sep 10 17:23:21.434393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:21.434363    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk"
Sep 10 17:23:59.997898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:23:59.997819    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-584cbdc55b-c779r"]
Sep 10 17:24:00.001434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.001409    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.004477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.004455    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 17:24:00.005235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.005215    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vbl9d\""
Sep 10 17:24:00.014811 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.014785    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584cbdc55b-c779r"]
Sep 10 17:24:00.084450 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.084413    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aeedba45-d8c6-44df-9182-e25748511025-tls-cert\") pod \"authorino-584cbdc55b-c779r\" (UID: \"aeedba45-d8c6-44df-9182-e25748511025\") " pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.084705 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.084458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh76x\" (UniqueName: \"kubernetes.io/projected/aeedba45-d8c6-44df-9182-e25748511025-kube-api-access-wh76x\") pod \"authorino-584cbdc55b-c779r\" (UID: \"aeedba45-d8c6-44df-9182-e25748511025\") " pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.185987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.185948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aeedba45-d8c6-44df-9182-e25748511025-tls-cert\") pod \"authorino-584cbdc55b-c779r\" (UID: \"aeedba45-d8c6-44df-9182-e25748511025\") " pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.185987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.185993    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh76x\" (UniqueName: \"kubernetes.io/projected/aeedba45-d8c6-44df-9182-e25748511025-kube-api-access-wh76x\") pod \"authorino-584cbdc55b-c779r\" (UID: \"aeedba45-d8c6-44df-9182-e25748511025\") " pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.188545 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.188523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/aeedba45-d8c6-44df-9182-e25748511025-tls-cert\") pod \"authorino-584cbdc55b-c779r\" (UID: \"aeedba45-d8c6-44df-9182-e25748511025\") " pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.194645 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.194622    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh76x\" (UniqueName: \"kubernetes.io/projected/aeedba45-d8c6-44df-9182-e25748511025-kube-api-access-wh76x\") pod \"authorino-584cbdc55b-c779r\" (UID: \"aeedba45-d8c6-44df-9182-e25748511025\") " pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.310597 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.310515    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584cbdc55b-c779r"
Sep 10 17:24:00.597535 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:00.597506    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584cbdc55b-c779r"]
Sep 10 17:24:00.599096 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:00.599070    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaeedba45_d8c6_44df_9182_e25748511025.slice/crio-0022438f6fb509101f247cf64ae507ed9ecceaf47102a12fdf651ccda2f887ac WatchSource:0}: Error finding container 0022438f6fb509101f247cf64ae507ed9ecceaf47102a12fdf651ccda2f887ac: Status 404 returned error can't find the container with id 0022438f6fb509101f247cf64ae507ed9ecceaf47102a12fdf651ccda2f887ac
Sep 10 17:24:01.605958 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:01.605904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584cbdc55b-c779r" event={"ID":"aeedba45-d8c6-44df-9182-e25748511025","Type":"ContainerStarted","Data":"2266be6f747255f6861610f20ace0a9d1909e0eb45a8e416d7e5cb0b31393016"}
Sep 10 17:24:01.606359 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:01.605977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584cbdc55b-c779r" event={"ID":"aeedba45-d8c6-44df-9182-e25748511025","Type":"ContainerStarted","Data":"0022438f6fb509101f247cf64ae507ed9ecceaf47102a12fdf651ccda2f887ac"}
Sep 10 17:24:01.638157 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:01.638093    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-584cbdc55b-c779r" podStartSLOduration=2.069731517 podStartE2EDuration="2.638073973s" totalImagesPullingTime="568.342456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:01.636058988 +0000 UTC m=+726.856182921" watchObservedRunningTime="2026-09-10 17:24:01.638073973 +0000 UTC m=+726.858197908"
Sep 10 17:24:31.203801 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.203762    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7650-d73ac7"]
Sep 10 17:24:31.207744 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.207725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:31.214330 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.214299    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4gkv4\""
Sep 10 17:24:31.230064 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.230029    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7650-d73ac7"]
Sep 10 17:24:31.278759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.278727    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxzf\" (UniqueName: \"kubernetes.io/projected/3bef8b39-47b5-4db6-9d65-6103d64d36f1-kube-api-access-8bxzf\") pod \"test-curl-7650-d73ac7\" (UID: \"3bef8b39-47b5-4db6-9d65-6103d64d36f1\") " pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:31.380225 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.380184    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxzf\" (UniqueName: \"kubernetes.io/projected/3bef8b39-47b5-4db6-9d65-6103d64d36f1-kube-api-access-8bxzf\") pod \"test-curl-7650-d73ac7\" (UID: \"3bef8b39-47b5-4db6-9d65-6103d64d36f1\") " pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:31.391064 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.391028    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxzf\" (UniqueName: \"kubernetes.io/projected/3bef8b39-47b5-4db6-9d65-6103d64d36f1-kube-api-access-8bxzf\") pod \"test-curl-7650-d73ac7\" (UID: \"3bef8b39-47b5-4db6-9d65-6103d64d36f1\") " pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:31.517267 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.517233    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:31.818468 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:31.818437    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bef8b39_47b5_4db6_9d65_6103d64d36f1.slice/crio-7cf80fb2da633c3b24ccf3fbdc2bf6082d729fec637e6b86725d67b866dcbfbe WatchSource:0}: Error finding container 7cf80fb2da633c3b24ccf3fbdc2bf6082d729fec637e6b86725d67b866dcbfbe: Status 404 returned error can't find the container with id 7cf80fb2da633c3b24ccf3fbdc2bf6082d729fec637e6b86725d67b866dcbfbe
Sep 10 17:24:31.822989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:31.822964    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7650-d73ac7"]
Sep 10 17:24:32.743191 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.743132    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"]
Sep 10 17:24:32.751178 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.751146    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:32.757303 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.757039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-dc5a37-dockercfg-hvtz4\""
Sep 10 17:24:32.757303 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.757114    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-dc5a37\""
Sep 10 17:24:32.771269 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.771228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7650-d73ac7" event={"ID":"3bef8b39-47b5-4db6-9d65-6103d64d36f1","Type":"ContainerStarted","Data":"7cf80fb2da633c3b24ccf3fbdc2bf6082d729fec637e6b86725d67b866dcbfbe"}
Sep 10 17:24:32.781048 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.781020    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"]
Sep 10 17:24:32.876846 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.876803    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"]
Sep 10 17:24:32.881140 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.881114    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:32.895307 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.895279    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6bc\" (UniqueName: \"kubernetes.io/projected/aef743e4-2063-4616-8692-998bb569290a-kube-api-access-rp6bc\") pod \"payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:32.895491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.895360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef743e4-2063-4616-8692-998bb569290a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:32.909879 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.909841    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"]
Sep 10 17:24:32.996246 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.996159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6bc\" (UniqueName: \"kubernetes.io/projected/aef743e4-2063-4616-8692-998bb569290a-kube-api-access-rp6bc\") pod \"payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:32.996246 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.996221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnx4g\" (UniqueName: \"kubernetes.io/projected/d8fde7a9-8230-4a81-b39e-36c51678dc23-kube-api-access-bnx4g\") pod \"payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:32.996461 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.996289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef743e4-2063-4616-8692-998bb569290a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:32.996461 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.996319    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fde7a9-8230-4a81-b39e-36c51678dc23-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:32.997155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:32.997131    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef743e4-2063-4616-8692-998bb569290a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:33.010678 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.010634    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6bc\" (UniqueName: \"kubernetes.io/projected/aef743e4-2063-4616-8692-998bb569290a-kube-api-access-rp6bc\") pod \"payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:33.064603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.064571    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:33.097345 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.097307    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnx4g\" (UniqueName: \"kubernetes.io/projected/d8fde7a9-8230-4a81-b39e-36c51678dc23-kube-api-access-bnx4g\") pod \"payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:33.097530 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.097400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fde7a9-8230-4a81-b39e-36c51678dc23-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:33.098191 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.098162    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fde7a9-8230-4a81-b39e-36c51678dc23-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:33.109111 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.109081    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnx4g\" (UniqueName: \"kubernetes.io/projected/d8fde7a9-8230-4a81-b39e-36c51678dc23-kube-api-access-bnx4g\") pod \"payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") " pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:33.194364 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.194330    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:33.358930 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:33.358897    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"]
Sep 10 17:24:33.783146 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:33.783104    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef743e4_2063_4616_8692_998bb569290a.slice/crio-4bac657cfdb8e044b2cd641322956b89fe2cf5662513cd8a214320a9565ddf62 WatchSource:0}: Error finding container 4bac657cfdb8e044b2cd641322956b89fe2cf5662513cd8a214320a9565ddf62: Status 404 returned error can't find the container with id 4bac657cfdb8e044b2cd641322956b89fe2cf5662513cd8a214320a9565ddf62
Sep 10 17:24:34.071868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.071710    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"]
Sep 10 17:24:34.074631 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:34.074596    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fde7a9_8230_4a81_b39e_36c51678dc23.slice/crio-ad5b56567c512179c52332d1cac8aa44e87923e827eec56cd710a12bc8227f22 WatchSource:0}: Error finding container ad5b56567c512179c52332d1cac8aa44e87923e827eec56cd710a12bc8227f22: Status 404 returned error can't find the container with id ad5b56567c512179c52332d1cac8aa44e87923e827eec56cd710a12bc8227f22
Sep 10 17:24:34.784927 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.784889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r" event={"ID":"aef743e4-2063-4616-8692-998bb569290a","Type":"ContainerStarted","Data":"daeac91883b482043bfd14123b14808e5504e43f1915ef8b625d2ca60cd1fabe"}
Sep 10 17:24:34.785433 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.784949    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r" event={"ID":"aef743e4-2063-4616-8692-998bb569290a","Type":"ContainerStarted","Data":"4bac657cfdb8e044b2cd641322956b89fe2cf5662513cd8a214320a9565ddf62"}
Sep 10 17:24:34.785433 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.785056    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:34.786646 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.786619    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7650-d73ac7" event={"ID":"3bef8b39-47b5-4db6-9d65-6103d64d36f1","Type":"ContainerStarted","Data":"fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe"}
Sep 10 17:24:34.788200 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.788179    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp" event={"ID":"d8fde7a9-8230-4a81-b39e-36c51678dc23","Type":"ContainerStarted","Data":"c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391"}
Sep 10 17:24:34.788292 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.788217    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp" event={"ID":"d8fde7a9-8230-4a81-b39e-36c51678dc23","Type":"ContainerStarted","Data":"ad5b56567c512179c52332d1cac8aa44e87923e827eec56cd710a12bc8227f22"}
Sep 10 17:24:34.788292 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.788236    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:34.813491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.813441    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r" podStartSLOduration=2.813424979 podStartE2EDuration="2.813424979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:34.813259637 +0000 UTC m=+760.033383571" watchObservedRunningTime="2026-09-10 17:24:34.813424979 +0000 UTC m=+760.033548906"
Sep 10 17:24:34.843203 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.843147    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp" podStartSLOduration=2.843127806 podStartE2EDuration="2.843127806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:34.841547654 +0000 UTC m=+760.061671590" watchObservedRunningTime="2026-09-10 17:24:34.843127806 +0000 UTC m=+760.063251744"
Sep 10 17:24:34.870436 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:34.870376    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7650-d73ac7" podStartSLOduration=1.8404186980000001 podStartE2EDuration="3.870359591s" totalImagesPullingTime="2.029940893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:34.868567047 +0000 UTC m=+760.088690981" watchObservedRunningTime="2026-09-10 17:24:34.870359591 +0000 UTC m=+760.090483524"
Sep 10 17:24:35.664778 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:35.664688    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7650-d73ac7"]
Sep 10 17:24:35.670917 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:35.670879    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7650-d73ac7"]
Sep 10 17:24:35.793995 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:24:35.793951    2556 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7650-d73ac7" container="test-curl-7650-d73ac7"
Sep 10 17:24:36.799484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:36.799431    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7650-d73ac7" podUID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" containerName="test-curl-7650-d73ac7" containerID="cri-o://fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe" gracePeriod=2
Sep 10 17:24:36.810682 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:24:36.809888    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7650-d73ac7\" is forbidden: User \"system:node:ip-10-0-137-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-19.ec2.internal' and this object" podUID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:38.980163 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:38.980133    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:39.055606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.055505    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bxzf\" (UniqueName: \"kubernetes.io/projected/3bef8b39-47b5-4db6-9d65-6103d64d36f1-kube-api-access-8bxzf\") pod \"3bef8b39-47b5-4db6-9d65-6103d64d36f1\" (UID: \"3bef8b39-47b5-4db6-9d65-6103d64d36f1\") "
Sep 10 17:24:39.057692 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.057632    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bef8b39-47b5-4db6-9d65-6103d64d36f1-kube-api-access-8bxzf" (OuterVolumeSpecName: "kube-api-access-8bxzf") pod "3bef8b39-47b5-4db6-9d65-6103d64d36f1" (UID: "3bef8b39-47b5-4db6-9d65-6103d64d36f1"). InnerVolumeSpecName "kube-api-access-8bxzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:24:39.156735 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.156698    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bxzf\" (UniqueName: \"kubernetes.io/projected/3bef8b39-47b5-4db6-9d65-6103d64d36f1-kube-api-access-8bxzf\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:24:39.272336 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.272300    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" path="/var/lib/kubelet/pods/3bef8b39-47b5-4db6-9d65-6103d64d36f1/volumes"
Sep 10 17:24:39.814417 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.814377    2556 generic.go:356] "Generic (PLEG): container finished" podID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" containerID="fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe" exitCode=137
Sep 10 17:24:39.814417 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.814419    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7650-d73ac7"
Sep 10 17:24:39.814730 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.814477    2556 scope.go:117] "RemoveContainer" containerID="fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe"
Sep 10 17:24:39.823192 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.823169    2556 scope.go:117] "RemoveContainer" containerID="fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe"
Sep 10 17:24:39.823477 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:24:39.823454    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe\": container with ID starting with fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe not found: ID does not exist" containerID="fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe"
Sep 10 17:24:39.823523 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:39.823486    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe"} err="failed to get container status \"fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe\": rpc error: code = NotFound desc = could not find container \"fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe\": container with ID starting with fd027178e092206835af4296ff5b1473032cf73656e566e2548f3302620ca9fe not found: ID does not exist"
Sep 10 17:24:44.374309 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.374271    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"]
Sep 10 17:24:44.374952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.374930    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" containerName="test-curl-7650-d73ac7"
Sep 10 17:24:44.375036 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.374954    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" containerName="test-curl-7650-d73ac7"
Sep 10 17:24:44.375089 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.375049    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bef8b39-47b5-4db6-9d65-6103d64d36f1" containerName="test-curl-7650-d73ac7"
Sep 10 17:24:44.378364 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.378342    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.382716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.382692    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-99ff0092\""
Sep 10 17:24:44.383185 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.383169    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-99ff0092-dockercfg-6zjkb\""
Sep 10 17:24:44.401489 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.401464    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"]
Sep 10 17:24:44.419016 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.418990    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"]
Sep 10 17:24:44.423152 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.423136    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.436502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.436479    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"]
Sep 10 17:24:44.517679 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.517637    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srqs2\" (UniqueName: \"kubernetes.io/projected/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-kube-api-access-srqs2\") pod \"payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.517866 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.517808    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a491bcb-4c59-4744-9411-8a4b61d43b54-plugins-config-volume\") pod \"payload-processing-e2e-ait-99ff0092-7589b9845-wb64x\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") " pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.517866 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.517849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzljt\" (UniqueName: \"kubernetes.io/projected/8a491bcb-4c59-4744-9411-8a4b61d43b54-kube-api-access-nzljt\") pod \"payload-processing-e2e-ait-99ff0092-7589b9845-wb64x\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") " pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.518032 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.517951    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.618989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.618948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzljt\" (UniqueName: \"kubernetes.io/projected/8a491bcb-4c59-4744-9411-8a4b61d43b54-kube-api-access-nzljt\") pod \"payload-processing-e2e-ait-99ff0092-7589b9845-wb64x\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") " pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.619153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.619019    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.619153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.619046    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srqs2\" (UniqueName: \"kubernetes.io/projected/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-kube-api-access-srqs2\") pod \"payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.619153 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.619128    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a491bcb-4c59-4744-9411-8a4b61d43b54-plugins-config-volume\") pod \"payload-processing-e2e-ait-99ff0092-7589b9845-wb64x\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") " pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.620005 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.619981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a491bcb-4c59-4744-9411-8a4b61d43b54-plugins-config-volume\") pod \"payload-processing-e2e-ait-99ff0092-7589b9845-wb64x\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") " pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.620055 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.619981    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.638241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.638180    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzljt\" (UniqueName: \"kubernetes.io/projected/8a491bcb-4c59-4744-9411-8a4b61d43b54-kube-api-access-nzljt\") pod \"payload-processing-e2e-ait-99ff0092-7589b9845-wb64x\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") " pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.645538 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.645513    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srqs2\" (UniqueName: \"kubernetes.io/projected/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-kube-api-access-srqs2\") pod \"payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.688423 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.688395    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:44.733001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.732965    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:44.975908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:44.975879    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"]
Sep 10 17:24:44.977282 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:44.977251    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04e5c78_21b4_4a7b_be9a_654db3e3b8f4.slice/crio-23c036755e2712fcb124bad9fff1bb9120da96295782755b6d24b78cba259444 WatchSource:0}: Error finding container 23c036755e2712fcb124bad9fff1bb9120da96295782755b6d24b78cba259444: Status 404 returned error can't find the container with id 23c036755e2712fcb124bad9fff1bb9120da96295782755b6d24b78cba259444
Sep 10 17:24:45.046772 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.046743    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"]
Sep 10 17:24:45.049318 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:45.049274    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a491bcb_4c59_4744_9411_8a4b61d43b54.slice/crio-359d587e48c5799fe9f4ff97b194c6d01aead727de26af4dd4a1e24bbdeb87d4 WatchSource:0}: Error finding container 359d587e48c5799fe9f4ff97b194c6d01aead727de26af4dd4a1e24bbdeb87d4: Status 404 returned error can't find the container with id 359d587e48c5799fe9f4ff97b194c6d01aead727de26af4dd4a1e24bbdeb87d4
Sep 10 17:24:45.796247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.796212    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:24:45.796629 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.796295    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:24:45.849017 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.848966    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6" event={"ID":"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4","Type":"ContainerStarted","Data":"11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0"}
Sep 10 17:24:45.849215 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.849045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6" event={"ID":"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4","Type":"ContainerStarted","Data":"23c036755e2712fcb124bad9fff1bb9120da96295782755b6d24b78cba259444"}
Sep 10 17:24:45.849215 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.849071    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:45.851013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.850977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x" event={"ID":"8a491bcb-4c59-4744-9411-8a4b61d43b54","Type":"ContainerStarted","Data":"6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c"}
Sep 10 17:24:45.851170 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.851040    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x" event={"ID":"8a491bcb-4c59-4744-9411-8a4b61d43b54","Type":"ContainerStarted","Data":"359d587e48c5799fe9f4ff97b194c6d01aead727de26af4dd4a1e24bbdeb87d4"}
Sep 10 17:24:45.851170 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.851127    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:45.919162 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.919106    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6" podStartSLOduration=1.9190878 podStartE2EDuration="1.9190878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:45.918051902 +0000 UTC m=+771.138175837" watchObservedRunningTime="2026-09-10 17:24:45.9190878 +0000 UTC m=+771.139211732"
Sep 10 17:24:45.948397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:45.948331    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x" podStartSLOduration=1.948308376 podStartE2EDuration="1.948308376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:45.943199122 +0000 UTC m=+771.163323055" watchObservedRunningTime="2026-09-10 17:24:45.948308376 +0000 UTC m=+771.168432313"
Sep 10 17:24:46.013085 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.013042    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"]
Sep 10 17:24:46.016763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.016746    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.020394 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.020369    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-889898-dockercfg-m6h5p\""
Sep 10 17:24:46.020536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.020372    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-889898\""
Sep 10 17:24:46.034179 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.034149    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"]
Sep 10 17:24:46.061143 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.061050    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"]
Sep 10 17:24:46.064757 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.064736    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.078833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.078805    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"]
Sep 10 17:24:46.134146 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.134102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c5fc89-e72c-4db9-98c8-35d16ab7d284-plugins-config-volume\") pod \"payload-processing-e2e-autores-889898-6979449c49-qnzb4\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") " pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.134146 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.134149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c9bs\" (UniqueName: \"kubernetes.io/projected/99c5fc89-e72c-4db9-98c8-35d16ab7d284-kube-api-access-4c9bs\") pod \"payload-processing-e2e-autores-889898-6979449c49-qnzb4\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") " pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.235050 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.234997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c5fc89-e72c-4db9-98c8-35d16ab7d284-plugins-config-volume\") pod \"payload-processing-e2e-autores-889898-6979449c49-qnzb4\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") " pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.235050 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.235048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c9bs\" (UniqueName: \"kubernetes.io/projected/99c5fc89-e72c-4db9-98c8-35d16ab7d284-kube-api-access-4c9bs\") pod \"payload-processing-e2e-autores-889898-6979449c49-qnzb4\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") " pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.235325 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.235159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jd2\" (UniqueName: \"kubernetes.io/projected/796252e7-f3da-4037-8993-67c6400cc05e-kube-api-access-96jd2\") pod \"payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.235325 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.235248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796252e7-f3da-4037-8993-67c6400cc05e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.235827 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.235803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c5fc89-e72c-4db9-98c8-35d16ab7d284-plugins-config-volume\") pod \"payload-processing-e2e-autores-889898-6979449c49-qnzb4\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") " pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.245506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.245477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c9bs\" (UniqueName: \"kubernetes.io/projected/99c5fc89-e72c-4db9-98c8-35d16ab7d284-kube-api-access-4c9bs\") pod \"payload-processing-e2e-autores-889898-6979449c49-qnzb4\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") " pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.334593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.334493    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.335868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.335843    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jd2\" (UniqueName: \"kubernetes.io/projected/796252e7-f3da-4037-8993-67c6400cc05e-kube-api-access-96jd2\") pod \"payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.335945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.335904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796252e7-f3da-4037-8993-67c6400cc05e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.336487 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.336469    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796252e7-f3da-4037-8993-67c6400cc05e-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.345880 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.345859    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jd2\" (UniqueName: \"kubernetes.io/projected/796252e7-f3da-4037-8993-67c6400cc05e-kube-api-access-96jd2\") pod \"payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.378460 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.378347    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.618246 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.618206    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"]
Sep 10 17:24:46.620356 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:46.620327    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c5fc89_e72c_4db9_98c8_35d16ab7d284.slice/crio-86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2 WatchSource:0}: Error finding container 86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2: Status 404 returned error can't find the container with id 86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2
Sep 10 17:24:46.673997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.673973    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"]
Sep 10 17:24:46.679577 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:24:46.679539    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796252e7_f3da_4037_8993_67c6400cc05e.slice/crio-a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7 WatchSource:0}: Error finding container a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7: Status 404 returned error can't find the container with id a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7
Sep 10 17:24:46.858259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.858220    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm" event={"ID":"796252e7-f3da-4037-8993-67c6400cc05e","Type":"ContainerStarted","Data":"666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8"}
Sep 10 17:24:46.858761 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.858287    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:24:46.858761 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.858311    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm" event={"ID":"796252e7-f3da-4037-8993-67c6400cc05e","Type":"ContainerStarted","Data":"a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7"}
Sep 10 17:24:46.860011 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.859981    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4" event={"ID":"99c5fc89-e72c-4db9-98c8-35d16ab7d284","Type":"ContainerStarted","Data":"93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88"}
Sep 10 17:24:46.860113 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.860036    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4" event={"ID":"99c5fc89-e72c-4db9-98c8-35d16ab7d284","Type":"ContainerStarted","Data":"86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2"}
Sep 10 17:24:46.860176 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.860124    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:46.888592 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.888487    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm" podStartSLOduration=0.88847211 podStartE2EDuration="888.47211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:46.886586231 +0000 UTC m=+772.106710168" watchObservedRunningTime="2026-09-10 17:24:46.88847211 +0000 UTC m=+772.108596042"
Sep 10 17:24:46.915173 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:46.915117    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4" podStartSLOduration=1.915099649 podStartE2EDuration="1.915099649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:24:46.912956058 +0000 UTC m=+772.133079991" watchObservedRunningTime="2026-09-10 17:24:46.915099649 +0000 UTC m=+772.135223587"
Sep 10 17:24:56.862072 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:56.862037    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:24:56.862461 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:56.862119    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:24:57.867799 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:57.867764    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:24:57.868187 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:24:57.867941    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:25:05.744877 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.744836    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"]
Sep 10 17:25:05.750091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.750061    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:05.755868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.755836    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-75bd9a\""
Sep 10 17:25:05.756016 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.755935    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-75bd9a-dockercfg-gn6rh\""
Sep 10 17:25:05.762796 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.762760    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"]
Sep 10 17:25:05.808282 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.808251    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"]
Sep 10 17:25:05.811887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.811870    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:05.825522 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.825497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"]
Sep 10 17:25:05.828868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.828844    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztvg\" (UniqueName: \"kubernetes.io/projected/4caf9198-044f-407a-aa04-770d7f858b7b-kube-api-access-6ztvg\") pod \"payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:05.828971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.828880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caf9198-044f-407a-aa04-770d7f858b7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:05.930084 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.930040    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztvg\" (UniqueName: \"kubernetes.io/projected/4caf9198-044f-407a-aa04-770d7f858b7b-kube-api-access-6ztvg\") pod \"payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:05.930084 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.930090    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caf9198-044f-407a-aa04-770d7f858b7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:05.930323 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.930127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b269482-1d79-4c96-9835-afd17f10bfdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:05.930323 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.930194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sz7w\" (UniqueName: \"kubernetes.io/projected/6b269482-1d79-4c96-9835-afd17f10bfdb-kube-api-access-6sz7w\") pod \"payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:05.930876 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.930854    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caf9198-044f-407a-aa04-770d7f858b7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:05.964763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:05.964730    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztvg\" (UniqueName: \"kubernetes.io/projected/4caf9198-044f-407a-aa04-770d7f858b7b-kube-api-access-6ztvg\") pod \"payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:06.031057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.030951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sz7w\" (UniqueName: \"kubernetes.io/projected/6b269482-1d79-4c96-9835-afd17f10bfdb-kube-api-access-6sz7w\") pod \"payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:06.031239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.031067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b269482-1d79-4c96-9835-afd17f10bfdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:06.031751 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.031731    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b269482-1d79-4c96-9835-afd17f10bfdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:06.057299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.057252    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sz7w\" (UniqueName: \"kubernetes.io/projected/6b269482-1d79-4c96-9835-afd17f10bfdb-kube-api-access-6sz7w\") pod \"payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:06.061108 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.061084    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:06.123208 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.123008    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:06.394762 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.394658    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"]
Sep 10 17:25:06.403652 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:25:06.403618    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caf9198_044f_407a_aa04_770d7f858b7b.slice/crio-8397dec901ce23db363944f3832bfc2a9f7c6c6762e980ed773e83cafd41e800 WatchSource:0}: Error finding container 8397dec901ce23db363944f3832bfc2a9f7c6c6762e980ed773e83cafd41e800: Status 404 returned error can't find the container with id 8397dec901ce23db363944f3832bfc2a9f7c6c6762e980ed773e83cafd41e800
Sep 10 17:25:06.485923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.485893    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"]
Sep 10 17:25:06.489168 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:25:06.489130    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b269482_1d79_4c96_9835_afd17f10bfdb.slice/crio-03d42a13f7bb319e66be8c8969660f9b6b693fafac96e06af26e94d62382177b WatchSource:0}: Error finding container 03d42a13f7bb319e66be8c8969660f9b6b693fafac96e06af26e94d62382177b: Status 404 returned error can't find the container with id 03d42a13f7bb319e66be8c8969660f9b6b693fafac96e06af26e94d62382177b
Sep 10 17:25:06.973921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.973765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5" event={"ID":"4caf9198-044f-407a-aa04-770d7f858b7b","Type":"ContainerStarted","Data":"1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927"}
Sep 10 17:25:06.973921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.973839    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5" event={"ID":"4caf9198-044f-407a-aa04-770d7f858b7b","Type":"ContainerStarted","Data":"8397dec901ce23db363944f3832bfc2a9f7c6c6762e980ed773e83cafd41e800"}
Sep 10 17:25:06.974452 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.974015    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:06.976116 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.976087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb" event={"ID":"6b269482-1d79-4c96-9835-afd17f10bfdb","Type":"ContainerStarted","Data":"3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165"}
Sep 10 17:25:06.976235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.976140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb" event={"ID":"6b269482-1d79-4c96-9835-afd17f10bfdb","Type":"ContainerStarted","Data":"03d42a13f7bb319e66be8c8969660f9b6b693fafac96e06af26e94d62382177b"}
Sep 10 17:25:06.976235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.976168    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:06.998894 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:06.998837    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5" podStartSLOduration=1.9988195370000001 podStartE2EDuration="1.998819537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:06.997211949 +0000 UTC m=+792.217335874" watchObservedRunningTime="2026-09-10 17:25:06.998819537 +0000 UTC m=+792.218943469"
Sep 10 17:25:07.035764 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:07.035706    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb" podStartSLOduration=2.035656084 podStartE2EDuration="2.035656084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:07.0336542 +0000 UTC m=+792.253778134" watchObservedRunningTime="2026-09-10 17:25:07.035656084 +0000 UTC m=+792.255780017"
Sep 10 17:25:17.984648 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:17.984566    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:25:17.984648 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:17.984642    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:25:24.216784 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.216742    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"]
Sep 10 17:25:24.217242 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.216972    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x" podUID="8a491bcb-4c59-4744-9411-8a4b61d43b54" containerName="payload-processing" containerID="cri-o://6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c" gracePeriod=30
Sep 10 17:25:24.231967 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.231937    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"]
Sep 10 17:25:24.232180 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.232156    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6" podUID="c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" containerName="payload-pre-processing" containerID="cri-o://11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0" gracePeriod=30
Sep 10 17:25:24.508166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.508139    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:25:24.511367 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.511348    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:25:24.614950 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.614910    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srqs2\" (UniqueName: \"kubernetes.io/projected/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-kube-api-access-srqs2\") pod \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") "
Sep 10 17:25:24.614950 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.614958    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a491bcb-4c59-4744-9411-8a4b61d43b54-plugins-config-volume\") pod \"8a491bcb-4c59-4744-9411-8a4b61d43b54\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") "
Sep 10 17:25:24.615210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.614977    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-plugins-config-volume\") pod \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\" (UID: \"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4\") "
Sep 10 17:25:24.615210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.615018    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzljt\" (UniqueName: \"kubernetes.io/projected/8a491bcb-4c59-4744-9411-8a4b61d43b54-kube-api-access-nzljt\") pod \"8a491bcb-4c59-4744-9411-8a4b61d43b54\" (UID: \"8a491bcb-4c59-4744-9411-8a4b61d43b54\") "
Sep 10 17:25:24.615476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.615445    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" (UID: "c04e5c78-21b4-4a7b-be9a-654db3e3b8f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:25:24.615549 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.615522    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a491bcb-4c59-4744-9411-8a4b61d43b54-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a491bcb-4c59-4744-9411-8a4b61d43b54" (UID: "8a491bcb-4c59-4744-9411-8a4b61d43b54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:25:24.621828 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.621793    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-kube-api-access-srqs2" (OuterVolumeSpecName: "kube-api-access-srqs2") pod "c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" (UID: "c04e5c78-21b4-4a7b-be9a-654db3e3b8f4"). InnerVolumeSpecName "kube-api-access-srqs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:25:24.621966 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.621839    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a491bcb-4c59-4744-9411-8a4b61d43b54-kube-api-access-nzljt" (OuterVolumeSpecName: "kube-api-access-nzljt") pod "8a491bcb-4c59-4744-9411-8a4b61d43b54" (UID: "8a491bcb-4c59-4744-9411-8a4b61d43b54"). InnerVolumeSpecName "kube-api-access-nzljt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:25:24.716368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.716329    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srqs2\" (UniqueName: \"kubernetes.io/projected/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-kube-api-access-srqs2\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:25:24.716368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.716360    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a491bcb-4c59-4744-9411-8a4b61d43b54-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:25:24.716368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.716369    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:25:24.716368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:24.716377    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzljt\" (UniqueName: \"kubernetes.io/projected/8a491bcb-4c59-4744-9411-8a4b61d43b54-kube-api-access-nzljt\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:25:25.073854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.073816    2556 generic.go:356] "Generic (PLEG): container finished" podID="8a491bcb-4c59-4744-9411-8a4b61d43b54" containerID="6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c" exitCode=0
Sep 10 17:25:25.074058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.073873    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"
Sep 10 17:25:25.074058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.073913    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x" event={"ID":"8a491bcb-4c59-4744-9411-8a4b61d43b54","Type":"ContainerDied","Data":"6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c"}
Sep 10 17:25:25.074058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.073974    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x" event={"ID":"8a491bcb-4c59-4744-9411-8a4b61d43b54","Type":"ContainerDied","Data":"359d587e48c5799fe9f4ff97b194c6d01aead727de26af4dd4a1e24bbdeb87d4"}
Sep 10 17:25:25.074058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.074006    2556 scope.go:117] "RemoveContainer" containerID="6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c"
Sep 10 17:25:25.075569 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.075541    2556 generic.go:356] "Generic (PLEG): container finished" podID="c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" containerID="11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0" exitCode=0
Sep 10 17:25:25.075711 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.075577    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"
Sep 10 17:25:25.075711 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.075620    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6" event={"ID":"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4","Type":"ContainerDied","Data":"11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0"}
Sep 10 17:25:25.075711 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.075684    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6" event={"ID":"c04e5c78-21b4-4a7b-be9a-654db3e3b8f4","Type":"ContainerDied","Data":"23c036755e2712fcb124bad9fff1bb9120da96295782755b6d24b78cba259444"}
Sep 10 17:25:25.086850 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.086732    2556 scope.go:117] "RemoveContainer" containerID="6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c"
Sep 10 17:25:25.087136 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:25:25.087103    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c\": container with ID starting with 6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c not found: ID does not exist" containerID="6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c"
Sep 10 17:25:25.087200 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.087135    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c"} err="failed to get container status \"6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c\": rpc error: code = NotFound desc = could not find container \"6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c\": container with ID starting with 6129134179ef04554dc21ac9d90108bda2d951a917ed6ebd34738dfc81b5f04c not found: ID does not exist"
Sep 10 17:25:25.087200 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.087154    2556 scope.go:117] "RemoveContainer" containerID="11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0"
Sep 10 17:25:25.095221 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.095203    2556 scope.go:117] "RemoveContainer" containerID="11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0"
Sep 10 17:25:25.095520 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:25:25.095500    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0\": container with ID starting with 11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0 not found: ID does not exist" containerID="11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0"
Sep 10 17:25:25.095592 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.095530    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0"} err="failed to get container status \"11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0\": rpc error: code = NotFound desc = could not find container \"11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0\": container with ID starting with 11b42f52f1da81e441c9655a9036f4666d921e70969c872a68933a943809bef0 not found: ID does not exist"
Sep 10 17:25:25.191514 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.191478    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"]
Sep 10 17:25:25.197739 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.197713    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-99ff0092-79445585bb-qbcj6"]
Sep 10 17:25:25.219389 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.219358    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"]
Sep 10 17:25:25.231216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.231191    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-99ff0092-7589b9845-wb64x"]
Sep 10 17:25:25.272080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.272046    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a491bcb-4c59-4744-9411-8a4b61d43b54" path="/var/lib/kubelet/pods/8a491bcb-4c59-4744-9411-8a4b61d43b54/volumes"
Sep 10 17:25:25.272511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:25.272490    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" path="/var/lib/kubelet/pods/c04e5c78-21b4-4a7b-be9a-654db3e3b8f4/volumes"
Sep 10 17:25:29.924070 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924028    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"]
Sep 10 17:25:29.924471 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924458    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" containerName="payload-pre-processing"
Sep 10 17:25:29.924511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924474    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" containerName="payload-pre-processing"
Sep 10 17:25:29.924511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924500    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a491bcb-4c59-4744-9411-8a4b61d43b54" containerName="payload-processing"
Sep 10 17:25:29.924511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924505    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a491bcb-4c59-4744-9411-8a4b61d43b54" containerName="payload-processing"
Sep 10 17:25:29.924622 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924573    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a491bcb-4c59-4744-9411-8a4b61d43b54" containerName="payload-processing"
Sep 10 17:25:29.924622 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.924581    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04e5c78-21b4-4a7b-be9a-654db3e3b8f4" containerName="payload-pre-processing"
Sep 10 17:25:29.929442 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.929425    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:29.934139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.934117    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-53de5224\""
Sep 10 17:25:29.935173 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.935150    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-53de5224-dockercfg-f8rsl\""
Sep 10 17:25:29.948035 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.948012    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"]
Sep 10 17:25:29.959998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.959971    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d93530-4125-44c1-aeb6-ba981f0662c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:29.960140 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:29.960003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngkt\" (UniqueName: \"kubernetes.io/projected/23d93530-4125-44c1-aeb6-ba981f0662c9-kube-api-access-fngkt\") pod \"payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:30.060746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:30.060711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d93530-4125-44c1-aeb6-ba981f0662c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:30.060955 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:30.060823    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngkt\" (UniqueName: \"kubernetes.io/projected/23d93530-4125-44c1-aeb6-ba981f0662c9-kube-api-access-fngkt\") pod \"payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:30.061467 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:30.061438    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d93530-4125-44c1-aeb6-ba981f0662c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:30.085967 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:30.085940    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngkt\" (UniqueName: \"kubernetes.io/projected/23d93530-4125-44c1-aeb6-ba981f0662c9-kube-api-access-fngkt\") pod \"payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:30.240208 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:30.240171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:30.570204 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:30.570176    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"]
Sep 10 17:25:30.572536 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:25:30.572506    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d93530_4125_44c1_aeb6_ba981f0662c9.slice/crio-34987acd62beabbe29d0cf9cfe5432141a21b813096598151ccf0404f700f5cb WatchSource:0}: Error finding container 34987acd62beabbe29d0cf9cfe5432141a21b813096598151ccf0404f700f5cb: Status 404 returned error can't find the container with id 34987acd62beabbe29d0cf9cfe5432141a21b813096598151ccf0404f700f5cb
Sep 10 17:25:31.109719 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:31.109679    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf" event={"ID":"23d93530-4125-44c1-aeb6-ba981f0662c9","Type":"ContainerStarted","Data":"5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d"}
Sep 10 17:25:31.110211 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:31.109739    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf" event={"ID":"23d93530-4125-44c1-aeb6-ba981f0662c9","Type":"ContainerStarted","Data":"34987acd62beabbe29d0cf9cfe5432141a21b813096598151ccf0404f700f5cb"}
Sep 10 17:25:31.110211 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:31.109781    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:42.117359 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:42.117325    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:25:42.143263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:42.143203    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf" podStartSLOduration=13.143179315 podStartE2EDuration="13.143179315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:25:31.170754341 +0000 UTC m=+816.390878274" watchObservedRunningTime="2026-09-10 17:25:42.143179315 +0000 UTC m=+827.363303251"
Sep 10 17:25:59.753675 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:59.753620    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"]
Sep 10 17:25:59.754183 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:25:59.753864    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf" podUID="23d93530-4125-44c1-aeb6-ba981f0662c9" containerName="payload-pre-processing" containerID="cri-o://5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d" gracePeriod=30
Sep 10 17:26:00.010400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.010324    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:26:00.066400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.066361    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d93530-4125-44c1-aeb6-ba981f0662c9-plugins-config-volume\") pod \"23d93530-4125-44c1-aeb6-ba981f0662c9\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") "
Sep 10 17:26:00.066582 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.066410    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fngkt\" (UniqueName: \"kubernetes.io/projected/23d93530-4125-44c1-aeb6-ba981f0662c9-kube-api-access-fngkt\") pod \"23d93530-4125-44c1-aeb6-ba981f0662c9\" (UID: \"23d93530-4125-44c1-aeb6-ba981f0662c9\") "
Sep 10 17:26:00.066765 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.066742    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d93530-4125-44c1-aeb6-ba981f0662c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23d93530-4125-44c1-aeb6-ba981f0662c9" (UID: "23d93530-4125-44c1-aeb6-ba981f0662c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:00.068644 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.068613    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d93530-4125-44c1-aeb6-ba981f0662c9-kube-api-access-fngkt" (OuterVolumeSpecName: "kube-api-access-fngkt") pod "23d93530-4125-44c1-aeb6-ba981f0662c9" (UID: "23d93530-4125-44c1-aeb6-ba981f0662c9"). InnerVolumeSpecName "kube-api-access-fngkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:00.167791 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.167748    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23d93530-4125-44c1-aeb6-ba981f0662c9-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:26:00.167791 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.167781    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngkt\" (UniqueName: \"kubernetes.io/projected/23d93530-4125-44c1-aeb6-ba981f0662c9-kube-api-access-fngkt\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:26:00.260242 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.260205    2556 generic.go:356] "Generic (PLEG): container finished" podID="23d93530-4125-44c1-aeb6-ba981f0662c9" containerID="5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d" exitCode=0
Sep 10 17:26:00.260453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.260254    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"
Sep 10 17:26:00.260453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.260290    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf" event={"ID":"23d93530-4125-44c1-aeb6-ba981f0662c9","Type":"ContainerDied","Data":"5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d"}
Sep 10 17:26:00.260453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.260347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf" event={"ID":"23d93530-4125-44c1-aeb6-ba981f0662c9","Type":"ContainerDied","Data":"34987acd62beabbe29d0cf9cfe5432141a21b813096598151ccf0404f700f5cb"}
Sep 10 17:26:00.260453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.260369    2556 scope.go:117] "RemoveContainer" containerID="5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d"
Sep 10 17:26:00.270579 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.270560    2556 scope.go:117] "RemoveContainer" containerID="5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d"
Sep 10 17:26:00.270859 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:26:00.270841    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d\": container with ID starting with 5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d not found: ID does not exist" containerID="5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d"
Sep 10 17:26:00.270932 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.270866    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d"} err="failed to get container status \"5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d\": rpc error: code = NotFound desc = could not find container \"5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d\": container with ID starting with 5eef79b7a7afdc9ba6e07e5c37f732f7d6226806df4d4f502ee6c43313ee157d not found: ID does not exist"
Sep 10 17:26:00.351015 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.350979    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"]
Sep 10 17:26:00.439406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:00.439359    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-53de5224-5d56d9c894-v4zjf"]
Sep 10 17:26:01.271602 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:01.271570    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d93530-4125-44c1-aeb6-ba981f0662c9" path="/var/lib/kubelet/pods/23d93530-4125-44c1-aeb6-ba981f0662c9/volumes"
Sep 10 17:26:09.722729 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.722692    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"]
Sep 10 17:26:09.723190 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.723173    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d93530-4125-44c1-aeb6-ba981f0662c9" containerName="payload-pre-processing"
Sep 10 17:26:09.723269 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.723192    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d93530-4125-44c1-aeb6-ba981f0662c9" containerName="payload-pre-processing"
Sep 10 17:26:09.723320 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.723291    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d93530-4125-44c1-aeb6-ba981f0662c9" containerName="payload-pre-processing"
Sep 10 17:26:09.727558 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.727535    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.730309 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.730281    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-889898\"/\"autores-model-889898-kserve-self-signed-certs\""
Sep 10 17:26:09.730428 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.730370    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-889898\"/\"default-dockercfg-c7gnm\""
Sep 10 17:26:09.730490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.730450    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-889898\"/\"kube-root-ca.crt\""
Sep 10 17:26:09.730806 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.730786    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-889898\"/\"openshift-service-ca.crt\""
Sep 10 17:26:09.751045 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.751017    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"]
Sep 10 17:26:09.756621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.756595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-tmp-dir\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.756738 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.756632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wrb\" (UniqueName: \"kubernetes.io/projected/9e19c17e-55ea-4c08-a686-150b95e2eab3-kube-api-access-67wrb\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.756780 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.756739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-home\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.756780 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.756761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e19c17e-55ea-4c08-a686-150b95e2eab3-tls-certs\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.756842 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.756785    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-dshm\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.756842 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.756820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-model-cache\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-home\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e19c17e-55ea-4c08-a686-150b95e2eab3-tls-certs\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857685 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857466    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-dshm\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857685 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-model-cache\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857685 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857545    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-tmp-dir\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857685 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857568    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wrb\" (UniqueName: \"kubernetes.io/projected/9e19c17e-55ea-4c08-a686-150b95e2eab3-kube-api-access-67wrb\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.857906 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.857837    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-home\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.858027 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.858005    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-tmp-dir\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.858099 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.858008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-model-cache\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.859781 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.859755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-dshm\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.860069 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.860049    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e19c17e-55ea-4c08-a686-150b95e2eab3-tls-certs\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:09.874095 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:09.874068    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wrb\" (UniqueName: \"kubernetes.io/projected/9e19c17e-55ea-4c08-a686-150b95e2eab3-kube-api-access-67wrb\") pod \"autores-model-889898-kserve-78fcf56979-xpqjg\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") " pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:10.037359 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:10.037269    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:10.329481 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:10.329396    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"]
Sep 10 17:26:10.333207 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:26:10.333178    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e19c17e_55ea_4c08_a686_150b95e2eab3.slice/crio-4d87b9ba58aa554359f434a8bd4b10c1e254a747eb82c0f79b03061546ca83ce WatchSource:0}: Error finding container 4d87b9ba58aa554359f434a8bd4b10c1e254a747eb82c0f79b03061546ca83ce: Status 404 returned error can't find the container with id 4d87b9ba58aa554359f434a8bd4b10c1e254a747eb82c0f79b03061546ca83ce
Sep 10 17:26:11.322406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:11.322368    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg" event={"ID":"9e19c17e-55ea-4c08-a686-150b95e2eab3","Type":"ContainerStarted","Data":"84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f"}
Sep 10 17:26:11.322406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:11.322418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg" event={"ID":"9e19c17e-55ea-4c08-a686-150b95e2eab3","Type":"ContainerStarted","Data":"4d87b9ba58aa554359f434a8bd4b10c1e254a747eb82c0f79b03061546ca83ce"}
Sep 10 17:26:18.636467 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.636408    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg" podStartSLOduration=9.636391 podStartE2EDuration="9.636391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:11.357060531 +0000 UTC m=+856.577184464" watchObservedRunningTime="2026-09-10 17:26:18.636391 +0000 UTC m=+863.856514938"
Sep 10 17:26:18.636992 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.636551    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"]
Sep 10 17:26:18.640030 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.640006    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.644168 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.644143    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dc5a37\"/\"kube-root-ca.crt\""
Sep 10 17:26:18.644312 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.644157    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dc5a37\"/\"default-dockercfg-mqwdh\""
Sep 10 17:26:18.644554 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.644537    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-dc5a37\"/\"openshift-service-ca.crt\""
Sep 10 17:26:18.644749 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.644732    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-dc5a37\"/\"test-model-dc5a37-kserve-self-signed-certs\""
Sep 10 17:26:18.652235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.652187    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"]
Sep 10 17:26:18.746166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.746126    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-model-cache\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.746335 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.746192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tmp-dir\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.746335 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.746258    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tls-certs\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.746487 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.746465    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-home\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.746559 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.746541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2xm\" (UniqueName: \"kubernetes.io/projected/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-kube-api-access-5s2xm\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.746629 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.746615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-dshm\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.847745 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.847694    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tls-certs\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.847967 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.847881    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-home\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.847967 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.847947    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2xm\" (UniqueName: \"kubernetes.io/projected/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-kube-api-access-5s2xm\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.848085 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.847992    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-dshm\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.848085 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.848036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-model-cache\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.848085 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.848073    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tmp-dir\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.848337 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.848304    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-home\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.848463 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.848426    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tmp-dir\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.848505 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.848477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-model-cache\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.850162 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.850143    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-dshm\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.850326 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.850308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tls-certs\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.859463 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.859428    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2xm\" (UniqueName: \"kubernetes.io/projected/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-kube-api-access-5s2xm\") pod \"test-model-dc5a37-kserve-6cb4f67995-s7rwh\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") " pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:18.953236 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:18.953141    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:19.240137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:19.240107    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"]
Sep 10 17:26:19.242467 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:26:19.242434    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11b9e423_fd84_4b62_b1ca_7f4a4a826d58.slice/crio-e10419fa047ad73485ce46d6746a0cb587832db9d86ebebfb00600c020933e47 WatchSource:0}: Error finding container e10419fa047ad73485ce46d6746a0cb587832db9d86ebebfb00600c020933e47: Status 404 returned error can't find the container with id e10419fa047ad73485ce46d6746a0cb587832db9d86ebebfb00600c020933e47
Sep 10 17:26:19.367276 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:19.367232    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh" event={"ID":"11b9e423-fd84-4b62-b1ca-7f4a4a826d58","Type":"ContainerStarted","Data":"f45d59e636d3049ebc1fe0a9d02bfb7d66d5d7004932edccb387d6c1392b1c09"}
Sep 10 17:26:19.367429 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:19.367284    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh" event={"ID":"11b9e423-fd84-4b62-b1ca-7f4a4a826d58","Type":"ContainerStarted","Data":"e10419fa047ad73485ce46d6746a0cb587832db9d86ebebfb00600c020933e47"}
Sep 10 17:26:19.390134 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:19.390084    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh" podStartSLOduration=1.390064951 podStartE2EDuration="1.390064951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:26:19.387648215 +0000 UTC m=+864.607772148" watchObservedRunningTime="2026-09-10 17:26:19.390064951 +0000 UTC m=+864.610188919"
Sep 10 17:26:20.037758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:20.037713    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:20.038184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:20.037778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:20.054840 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:20.054807    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:20.383981 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:20.383901    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:26:28.953759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:28.953720    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:28.954243 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:28.954006    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:28.966399 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:28.966370    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:29.431177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:29.431141    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:26:56.587726 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:56.587643    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"]
Sep 10 17:26:56.588213 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:56.587964    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4" podUID="99c5fc89-e72c-4db9-98c8-35d16ab7d284" containerName="payload-processing" containerID="cri-o://93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88" gracePeriod=30
Sep 10 17:26:56.595890 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:56.595864    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"]
Sep 10 17:26:56.596060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:56.596040    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm" podUID="796252e7-f3da-4037-8993-67c6400cc05e" containerName="payload-pre-processing" containerID="cri-o://666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8" gracePeriod=30
Sep 10 17:26:56.882769 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:56.882732    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:26:56.885107 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:56.885084    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:26:57.021635 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.021601    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796252e7-f3da-4037-8993-67c6400cc05e-plugins-config-volume\") pod \"796252e7-f3da-4037-8993-67c6400cc05e\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") "
Sep 10 17:26:57.021830 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.021735    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96jd2\" (UniqueName: \"kubernetes.io/projected/796252e7-f3da-4037-8993-67c6400cc05e-kube-api-access-96jd2\") pod \"796252e7-f3da-4037-8993-67c6400cc05e\" (UID: \"796252e7-f3da-4037-8993-67c6400cc05e\") "
Sep 10 17:26:57.021830 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.021794    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c9bs\" (UniqueName: \"kubernetes.io/projected/99c5fc89-e72c-4db9-98c8-35d16ab7d284-kube-api-access-4c9bs\") pod \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") "
Sep 10 17:26:57.021921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.021837    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c5fc89-e72c-4db9-98c8-35d16ab7d284-plugins-config-volume\") pod \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\" (UID: \"99c5fc89-e72c-4db9-98c8-35d16ab7d284\") "
Sep 10 17:26:57.022171 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.022057    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/796252e7-f3da-4037-8993-67c6400cc05e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "796252e7-f3da-4037-8993-67c6400cc05e" (UID: "796252e7-f3da-4037-8993-67c6400cc05e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:57.022291 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.022242    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c5fc89-e72c-4db9-98c8-35d16ab7d284-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99c5fc89-e72c-4db9-98c8-35d16ab7d284" (UID: "99c5fc89-e72c-4db9-98c8-35d16ab7d284"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:26:57.023853 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.023830    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c5fc89-e72c-4db9-98c8-35d16ab7d284-kube-api-access-4c9bs" (OuterVolumeSpecName: "kube-api-access-4c9bs") pod "99c5fc89-e72c-4db9-98c8-35d16ab7d284" (UID: "99c5fc89-e72c-4db9-98c8-35d16ab7d284"). InnerVolumeSpecName "kube-api-access-4c9bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:57.023943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.023860    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/796252e7-f3da-4037-8993-67c6400cc05e-kube-api-access-96jd2" (OuterVolumeSpecName: "kube-api-access-96jd2") pod "796252e7-f3da-4037-8993-67c6400cc05e" (UID: "796252e7-f3da-4037-8993-67c6400cc05e"). InnerVolumeSpecName "kube-api-access-96jd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:26:57.123424 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.123342    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/796252e7-f3da-4037-8993-67c6400cc05e-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:26:57.123424 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.123372    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96jd2\" (UniqueName: \"kubernetes.io/projected/796252e7-f3da-4037-8993-67c6400cc05e-kube-api-access-96jd2\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:26:57.123424 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.123382    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c9bs\" (UniqueName: \"kubernetes.io/projected/99c5fc89-e72c-4db9-98c8-35d16ab7d284-kube-api-access-4c9bs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:26:57.123424 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.123391    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c5fc89-e72c-4db9-98c8-35d16ab7d284-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:26:57.290190 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:26:57.290157    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796252e7_f3da_4037_8993_67c6400cc05e.slice/crio-a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c5fc89_e72c_4db9_98c8_35d16ab7d284.slice/crio-86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2\": RecentStats: unable to find data in memory cache]"
Sep 10 17:26:57.290341 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:26:57.290310    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c5fc89_e72c_4db9_98c8_35d16ab7d284.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c5fc89_e72c_4db9_98c8_35d16ab7d284.slice/crio-86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod796252e7_f3da_4037_8993_67c6400cc05e.slice/crio-a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7\": RecentStats: unable to find data in memory cache]"
Sep 10 17:26:57.571727 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.571689    2556 generic.go:356] "Generic (PLEG): container finished" podID="796252e7-f3da-4037-8993-67c6400cc05e" containerID="666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8" exitCode=0
Sep 10 17:26:57.571939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.571737    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"
Sep 10 17:26:57.571939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.571749    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm" event={"ID":"796252e7-f3da-4037-8993-67c6400cc05e","Type":"ContainerDied","Data":"666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8"}
Sep 10 17:26:57.571939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.571821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm" event={"ID":"796252e7-f3da-4037-8993-67c6400cc05e","Type":"ContainerDied","Data":"a1a7129172b9c72bec299b85e8349fc2528bad6b4a0297ba749fa148aee8f7f7"}
Sep 10 17:26:57.571939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.571843    2556 scope.go:117] "RemoveContainer" containerID="666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8"
Sep 10 17:26:57.573268 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.573244    2556 generic.go:356] "Generic (PLEG): container finished" podID="99c5fc89-e72c-4db9-98c8-35d16ab7d284" containerID="93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88" exitCode=0
Sep 10 17:26:57.573391 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.573320    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"
Sep 10 17:26:57.573391 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.573328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4" event={"ID":"99c5fc89-e72c-4db9-98c8-35d16ab7d284","Type":"ContainerDied","Data":"93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88"}
Sep 10 17:26:57.573391 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.573363    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4" event={"ID":"99c5fc89-e72c-4db9-98c8-35d16ab7d284","Type":"ContainerDied","Data":"86656f0b722833bf037fb6ef660e5dba5142ee1831a3ce35c9e19d13fb71aff2"}
Sep 10 17:26:57.581119 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.581102    2556 scope.go:117] "RemoveContainer" containerID="666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8"
Sep 10 17:26:57.581399 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:26:57.581378    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8\": container with ID starting with 666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8 not found: ID does not exist" containerID="666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8"
Sep 10 17:26:57.581471 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.581406    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8"} err="failed to get container status \"666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8\": rpc error: code = NotFound desc = could not find container \"666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8\": container with ID starting with 666453cf5d302d891a34d62ceab774ea0ff585b52d548fe0f5687c05213705a8 not found: ID does not exist"
Sep 10 17:26:57.581471 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.581423    2556 scope.go:117] "RemoveContainer" containerID="93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88"
Sep 10 17:26:57.589635 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.589478    2556 scope.go:117] "RemoveContainer" containerID="93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88"
Sep 10 17:26:57.590024 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:26:57.589747    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88\": container with ID starting with 93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88 not found: ID does not exist" containerID="93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88"
Sep 10 17:26:57.590024 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.589770    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88"} err="failed to get container status \"93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88\": rpc error: code = NotFound desc = could not find container \"93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88\": container with ID starting with 93b0135318a7dcb43d707bfe8643e3b6e2fbc750fbb624bc0cb8db4605c83c88 not found: ID does not exist"
Sep 10 17:26:57.591442 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.591424    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"]
Sep 10 17:26:57.593821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.593804    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-889898-c5b66d6-m2cwm"]
Sep 10 17:26:57.617992 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.617972    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"]
Sep 10 17:26:57.625647 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:57.625627    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-889898-6979449c49-qnzb4"]
Sep 10 17:26:59.272504 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:59.272472    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="796252e7-f3da-4037-8993-67c6400cc05e" path="/var/lib/kubelet/pods/796252e7-f3da-4037-8993-67c6400cc05e/volumes"
Sep 10 17:26:59.272891 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:26:59.272877    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c5fc89-e72c-4db9-98c8-35d16ab7d284" path="/var/lib/kubelet/pods/99c5fc89-e72c-4db9-98c8-35d16ab7d284/volumes"
Sep 10 17:27:02.213131 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:02.207330    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"]
Sep 10 17:27:07.852808 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.852772    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"]
Sep 10 17:27:07.853280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.853210    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="796252e7-f3da-4037-8993-67c6400cc05e" containerName="payload-pre-processing"
Sep 10 17:27:07.853280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.853222    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="796252e7-f3da-4037-8993-67c6400cc05e" containerName="payload-pre-processing"
Sep 10 17:27:07.853280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.853235    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c5fc89-e72c-4db9-98c8-35d16ab7d284" containerName="payload-processing"
Sep 10 17:27:07.853280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.853241    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c5fc89-e72c-4db9-98c8-35d16ab7d284" containerName="payload-processing"
Sep 10 17:27:07.853516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.853317    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c5fc89-e72c-4db9-98c8-35d16ab7d284" containerName="payload-processing"
Sep 10 17:27:07.853516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.853330    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="796252e7-f3da-4037-8993-67c6400cc05e" containerName="payload-pre-processing"
Sep 10 17:27:07.857963 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.857933    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:07.862698 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.862676    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-171e086f-kserve-self-signed-certs\""
Sep 10 17:27:07.875468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:07.875441    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"]
Sep 10 17:27:08.034946 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.034903    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-tmp-dir\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.035141 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.034973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-home\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.035141 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.035080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c3347-b92d-47c3-8b36-c49d3e990107-tls-certs\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.035233 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.035179    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-model-cache\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.035294 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.035232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-dshm\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.035355 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.035282    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzrbb\" (UniqueName: \"kubernetes.io/projected/c46c3347-b92d-47c3-8b36-c49d3e990107-kube-api-access-zzrbb\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.135992    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzrbb\" (UniqueName: \"kubernetes.io/projected/c46c3347-b92d-47c3-8b36-c49d3e990107-kube-api-access-zzrbb\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136036    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-tmp-dir\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136078    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-home\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136114    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c3347-b92d-47c3-8b36-c49d3e990107-tls-certs\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136165    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-model-cache\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136354 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-dshm\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136517 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136492    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-tmp-dir\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136559 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136494    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-home\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.136593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.136567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-model-cache\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.138516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.138489    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-dshm\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.138899 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.138872    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c3347-b92d-47c3-8b36-c49d3e990107-tls-certs\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.148379 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.148358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzrbb\" (UniqueName: \"kubernetes.io/projected/c46c3347-b92d-47c3-8b36-c49d3e990107-kube-api-access-zzrbb\") pod \"e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") " pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.167872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.167852    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:08.442833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.442809    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"]
Sep 10 17:27:08.443816 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:27:08.443786    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc46c3347_b92d_47c3_8b36_c49d3e990107.slice/crio-e1127ea755e8a72d570bdc70887d7de0d197308f4bee20b2364734ff22435703 WatchSource:0}: Error finding container e1127ea755e8a72d570bdc70887d7de0d197308f4bee20b2364734ff22435703: Status 404 returned error can't find the container with id e1127ea755e8a72d570bdc70887d7de0d197308f4bee20b2364734ff22435703
Sep 10 17:27:08.638164 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.638126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl" event={"ID":"c46c3347-b92d-47c3-8b36-c49d3e990107","Type":"ContainerStarted","Data":"01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7"}
Sep 10 17:27:08.638164 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.638169    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl" event={"ID":"c46c3347-b92d-47c3-8b36-c49d3e990107","Type":"ContainerStarted","Data":"e1127ea755e8a72d570bdc70887d7de0d197308f4bee20b2364734ff22435703"}
Sep 10 17:27:08.661678 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:08.661620    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl" podStartSLOduration=1.6616034100000001 podStartE2EDuration="1.66160341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:08.659475661 +0000 UTC m=+913.879599593" watchObservedRunningTime="2026-09-10 17:27:08.66160341 +0000 UTC m=+913.881727380"
Sep 10 17:27:16.636047 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:16.636005    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"]
Sep 10 17:27:17.232649 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.232596    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg" podUID="9e19c17e-55ea-4c08-a686-150b95e2eab3" containerName="main" containerID="cri-o://84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f" gracePeriod=45
Sep 10 17:27:17.495438 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.495373    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:27:17.528845 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.528817    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e19c17e-55ea-4c08-a686-150b95e2eab3-tls-certs\") pod \"9e19c17e-55ea-4c08-a686-150b95e2eab3\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") "
Sep 10 17:27:17.529016 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.528876    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-home\") pod \"9e19c17e-55ea-4c08-a686-150b95e2eab3\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") "
Sep 10 17:27:17.529076 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529020    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67wrb\" (UniqueName: \"kubernetes.io/projected/9e19c17e-55ea-4c08-a686-150b95e2eab3-kube-api-access-67wrb\") pod \"9e19c17e-55ea-4c08-a686-150b95e2eab3\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") "
Sep 10 17:27:17.529076 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529069    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-model-cache\") pod \"9e19c17e-55ea-4c08-a686-150b95e2eab3\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") "
Sep 10 17:27:17.529190 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529098    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-home" (OuterVolumeSpecName: "home") pod "9e19c17e-55ea-4c08-a686-150b95e2eab3" (UID: "9e19c17e-55ea-4c08-a686-150b95e2eab3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:17.529190 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529158    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-tmp-dir\") pod \"9e19c17e-55ea-4c08-a686-150b95e2eab3\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") "
Sep 10 17:27:17.529304 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-dshm\") pod \"9e19c17e-55ea-4c08-a686-150b95e2eab3\" (UID: \"9e19c17e-55ea-4c08-a686-150b95e2eab3\") "
Sep 10 17:27:17.529401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529372    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e19c17e-55ea-4c08-a686-150b95e2eab3" (UID: "9e19c17e-55ea-4c08-a686-150b95e2eab3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:17.529401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529385    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e19c17e-55ea-4c08-a686-150b95e2eab3" (UID: "9e19c17e-55ea-4c08-a686-150b95e2eab3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:17.529598 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529577    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.529598 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529602    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.529755 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.529619    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.531245 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.531207    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e19c17e-55ea-4c08-a686-150b95e2eab3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e19c17e-55ea-4c08-a686-150b95e2eab3" (UID: "9e19c17e-55ea-4c08-a686-150b95e2eab3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:17.531378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.531282    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e19c17e-55ea-4c08-a686-150b95e2eab3-kube-api-access-67wrb" (OuterVolumeSpecName: "kube-api-access-67wrb") pod "9e19c17e-55ea-4c08-a686-150b95e2eab3" (UID: "9e19c17e-55ea-4c08-a686-150b95e2eab3"). InnerVolumeSpecName "kube-api-access-67wrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:17.531453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.531408    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-dshm" (OuterVolumeSpecName: "dshm") pod "9e19c17e-55ea-4c08-a686-150b95e2eab3" (UID: "9e19c17e-55ea-4c08-a686-150b95e2eab3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:17.630092 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.630060    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wrb\" (UniqueName: \"kubernetes.io/projected/9e19c17e-55ea-4c08-a686-150b95e2eab3-kube-api-access-67wrb\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.630092 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.630086    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e19c17e-55ea-4c08-a686-150b95e2eab3-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.630092 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.630095    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e19c17e-55ea-4c08-a686-150b95e2eab3-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:17.689785 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.689744    2556 generic.go:356] "Generic (PLEG): container finished" podID="9e19c17e-55ea-4c08-a686-150b95e2eab3" containerID="84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f" exitCode=0
Sep 10 17:27:17.690228 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.689792    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"
Sep 10 17:27:17.690228 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.689802    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg" event={"ID":"9e19c17e-55ea-4c08-a686-150b95e2eab3","Type":"ContainerDied","Data":"84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f"}
Sep 10 17:27:17.690228 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.689842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg" event={"ID":"9e19c17e-55ea-4c08-a686-150b95e2eab3","Type":"ContainerDied","Data":"4d87b9ba58aa554359f434a8bd4b10c1e254a747eb82c0f79b03061546ca83ce"}
Sep 10 17:27:17.690228 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.689864    2556 scope.go:117] "RemoveContainer" containerID="84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f"
Sep 10 17:27:17.699792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.699774    2556 scope.go:117] "RemoveContainer" containerID="84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f"
Sep 10 17:27:17.700088 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:27:17.700072    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f\": container with ID starting with 84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f not found: ID does not exist" containerID="84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f"
Sep 10 17:27:17.700147 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.700097    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f"} err="failed to get container status \"84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f\": rpc error: code = NotFound desc = could not find container \"84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f\": container with ID starting with 84232c8d70098706be9ced14906d74454f4cb702766dc24e29b2c5a7be5f0d1f not found: ID does not exist"
Sep 10 17:27:17.713681 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.713646    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"]
Sep 10 17:27:17.720562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:17.720537    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-889898/autores-model-889898-kserve-78fcf56979-xpqjg"]
Sep 10 17:27:18.168896 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:18.168860    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:19.272410 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:19.272369    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e19c17e-55ea-4c08-a686-150b95e2eab3" path="/var/lib/kubelet/pods/9e19c17e-55ea-4c08-a686-150b95e2eab3/volumes"
Sep 10 17:27:31.656578 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.656520    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl" podUID="c46c3347-b92d-47c3-8b36-c49d3e990107" containerName="main" containerID="cri-o://01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7" gracePeriod=45
Sep 10 17:27:31.905355 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.905331    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:31.967501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967422    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-dshm\") pod \"c46c3347-b92d-47c3-8b36-c49d3e990107\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") "
Sep 10 17:27:31.967501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967459    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-tmp-dir\") pod \"c46c3347-b92d-47c3-8b36-c49d3e990107\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") "
Sep 10 17:27:31.967501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967478    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzrbb\" (UniqueName: \"kubernetes.io/projected/c46c3347-b92d-47c3-8b36-c49d3e990107-kube-api-access-zzrbb\") pod \"c46c3347-b92d-47c3-8b36-c49d3e990107\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") "
Sep 10 17:27:31.967792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967558    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-home\") pod \"c46c3347-b92d-47c3-8b36-c49d3e990107\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") "
Sep 10 17:27:31.967792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967656    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c3347-b92d-47c3-8b36-c49d3e990107-tls-certs\") pod \"c46c3347-b92d-47c3-8b36-c49d3e990107\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") "
Sep 10 17:27:31.967792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967706    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-model-cache\") pod \"c46c3347-b92d-47c3-8b36-c49d3e990107\" (UID: \"c46c3347-b92d-47c3-8b36-c49d3e990107\") "
Sep 10 17:27:31.967792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967727    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c46c3347-b92d-47c3-8b36-c49d3e990107" (UID: "c46c3347-b92d-47c3-8b36-c49d3e990107"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:31.968069 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.967835    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-home" (OuterVolumeSpecName: "home") pod "c46c3347-b92d-47c3-8b36-c49d3e990107" (UID: "c46c3347-b92d-47c3-8b36-c49d3e990107"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:31.968069 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.968054    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-model-cache" (OuterVolumeSpecName: "model-cache") pod "c46c3347-b92d-47c3-8b36-c49d3e990107" (UID: "c46c3347-b92d-47c3-8b36-c49d3e990107"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:31.968184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.968145    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:31.968184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.968158    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:31.968184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.968166    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:31.969786 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.969761    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c46c3347-b92d-47c3-8b36-c49d3e990107-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c46c3347-b92d-47c3-8b36-c49d3e990107" (UID: "c46c3347-b92d-47c3-8b36-c49d3e990107"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:27:31.969896 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.969791    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-dshm" (OuterVolumeSpecName: "dshm") pod "c46c3347-b92d-47c3-8b36-c49d3e990107" (UID: "c46c3347-b92d-47c3-8b36-c49d3e990107"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:27:31.969963 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:31.969909    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46c3347-b92d-47c3-8b36-c49d3e990107-kube-api-access-zzrbb" (OuterVolumeSpecName: "kube-api-access-zzrbb") pod "c46c3347-b92d-47c3-8b36-c49d3e990107" (UID: "c46c3347-b92d-47c3-8b36-c49d3e990107"). InnerVolumeSpecName "kube-api-access-zzrbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:27:32.069730 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.069690    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c46c3347-b92d-47c3-8b36-c49d3e990107-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:32.069730 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.069724    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzrbb\" (UniqueName: \"kubernetes.io/projected/c46c3347-b92d-47c3-8b36-c49d3e990107-kube-api-access-zzrbb\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:32.069730 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.069739    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c46c3347-b92d-47c3-8b36-c49d3e990107-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:27:32.777233 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.777197    2556 generic.go:356] "Generic (PLEG): container finished" podID="c46c3347-b92d-47c3-8b36-c49d3e990107" containerID="01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7" exitCode=0
Sep 10 17:27:32.777750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.777239    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"
Sep 10 17:27:32.777750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.777292    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl" event={"ID":"c46c3347-b92d-47c3-8b36-c49d3e990107","Type":"ContainerDied","Data":"01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7"}
Sep 10 17:27:32.777750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.777337    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl" event={"ID":"c46c3347-b92d-47c3-8b36-c49d3e990107","Type":"ContainerDied","Data":"e1127ea755e8a72d570bdc70887d7de0d197308f4bee20b2364734ff22435703"}
Sep 10 17:27:32.777750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.777357    2556 scope.go:117] "RemoveContainer" containerID="01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7"
Sep 10 17:27:32.788871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.788791    2556 scope.go:117] "RemoveContainer" containerID="01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7"
Sep 10 17:27:32.789289 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:27:32.789213    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7\": container with ID starting with 01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7 not found: ID does not exist" containerID="01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7"
Sep 10 17:27:32.789289 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.789250    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7"} err="failed to get container status \"01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7\": rpc error: code = NotFound desc = could not find container \"01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7\": container with ID starting with 01310b0a769ea10aac4402bb666cf89a699684199c63aa4dfd7a26af894d26e7 not found: ID does not exist"
Sep 10 17:27:32.801311 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.801289    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"]
Sep 10 17:27:32.803291 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:32.803270    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-171e086f-kserve-79c56c6d7c-9vhkl"]
Sep 10 17:27:33.271397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:33.271360    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c46c3347-b92d-47c3-8b36-c49d3e990107" path="/var/lib/kubelet/pods/c46c3347-b92d-47c3-8b36-c49d3e990107/volumes"
Sep 10 17:27:48.457499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.457461    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"]
Sep 10 17:27:48.459952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.457918    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e19c17e-55ea-4c08-a686-150b95e2eab3" containerName="main"
Sep 10 17:27:48.459952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.457931    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e19c17e-55ea-4c08-a686-150b95e2eab3" containerName="main"
Sep 10 17:27:48.459952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.457947    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46c3347-b92d-47c3-8b36-c49d3e990107" containerName="main"
Sep 10 17:27:48.459952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.457953    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46c3347-b92d-47c3-8b36-c49d3e990107" containerName="main"
Sep 10 17:27:48.459952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.458025    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c46c3347-b92d-47c3-8b36-c49d3e990107" containerName="main"
Sep 10 17:27:48.459952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.458037    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e19c17e-55ea-4c08-a686-150b95e2eab3" containerName="main"
Sep 10 17:27:48.460888 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.460872    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.470866 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.470843    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4b53b5-dockercfg-9dgcd\""
Sep 10 17:27:48.470989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.470880    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4b53b5\""
Sep 10 17:27:48.495230 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.495203    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"]
Sep 10 17:27:48.637041 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.637010    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcbe29-f650-41a6-929b-1e25f4e9a80f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.637041 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.637045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vl2\" (UniqueName: \"kubernetes.io/projected/59fcbe29-f650-41a6-929b-1e25f4e9a80f-kube-api-access-p7vl2\") pod \"payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.738317 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.738284    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcbe29-f650-41a6-929b-1e25f4e9a80f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.738477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.738333    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vl2\" (UniqueName: \"kubernetes.io/projected/59fcbe29-f650-41a6-929b-1e25f4e9a80f-kube-api-access-p7vl2\") pod \"payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.738996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.738975    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcbe29-f650-41a6-929b-1e25f4e9a80f-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.748966 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.748942    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vl2\" (UniqueName: \"kubernetes.io/projected/59fcbe29-f650-41a6-929b-1e25f4e9a80f-kube-api-access-p7vl2\") pod \"payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:48.771859 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:48.771804    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:49.047043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:49.047014    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"]
Sep 10 17:27:49.049779 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:27:49.049733    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fcbe29_f650_41a6_929b_1e25f4e9a80f.slice/crio-0cb3382c741625db512465f58ea313ca649c162d8b277f669e9bfe5174a46c2f WatchSource:0}: Error finding container 0cb3382c741625db512465f58ea313ca649c162d8b277f669e9bfe5174a46c2f: Status 404 returned error can't find the container with id 0cb3382c741625db512465f58ea313ca649c162d8b277f669e9bfe5174a46c2f
Sep 10 17:27:49.871371 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:49.871333    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn" event={"ID":"59fcbe29-f650-41a6-929b-1e25f4e9a80f","Type":"ContainerStarted","Data":"53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f"}
Sep 10 17:27:49.871792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:49.871393    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn" event={"ID":"59fcbe29-f650-41a6-929b-1e25f4e9a80f","Type":"ContainerStarted","Data":"0cb3382c741625db512465f58ea313ca649c162d8b277f669e9bfe5174a46c2f"}
Sep 10 17:27:49.871792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:49.871504    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:27:49.898074 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:27:49.898024    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn" podStartSLOduration=1.898005841 podStartE2EDuration="1.898005841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:27:49.89610893 +0000 UTC m=+955.116232864" watchObservedRunningTime="2026-09-10 17:27:49.898005841 +0000 UTC m=+955.118129774"
Sep 10 17:28:00.880033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:00.879998    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:28:50.024879 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:50.024838    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"]
Sep 10 17:28:52.166098 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.166061    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"]
Sep 10 17:28:52.166482 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.166272    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn" podUID="59fcbe29-f650-41a6-929b-1e25f4e9a80f" containerName="payload-pre-processing" containerID="cri-o://53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f" gracePeriod=30
Sep 10 17:28:52.428390 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.428321    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:28:52.571954 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.571918    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcbe29-f650-41a6-929b-1e25f4e9a80f-plugins-config-volume\") pod \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") "
Sep 10 17:28:52.572125 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.572022    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7vl2\" (UniqueName: \"kubernetes.io/projected/59fcbe29-f650-41a6-929b-1e25f4e9a80f-kube-api-access-p7vl2\") pod \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\" (UID: \"59fcbe29-f650-41a6-929b-1e25f4e9a80f\") "
Sep 10 17:28:52.572361 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.572333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fcbe29-f650-41a6-929b-1e25f4e9a80f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59fcbe29-f650-41a6-929b-1e25f4e9a80f" (UID: "59fcbe29-f650-41a6-929b-1e25f4e9a80f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:28:52.574088 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.574066    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fcbe29-f650-41a6-929b-1e25f4e9a80f-kube-api-access-p7vl2" (OuterVolumeSpecName: "kube-api-access-p7vl2") pod "59fcbe29-f650-41a6-929b-1e25f4e9a80f" (UID: "59fcbe29-f650-41a6-929b-1e25f4e9a80f"). InnerVolumeSpecName "kube-api-access-p7vl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:28:52.673043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.673006    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7vl2\" (UniqueName: \"kubernetes.io/projected/59fcbe29-f650-41a6-929b-1e25f4e9a80f-kube-api-access-p7vl2\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:28:52.673043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:52.673037    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59fcbe29-f650-41a6-929b-1e25f4e9a80f-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:28:53.207721 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.207657    2556 generic.go:356] "Generic (PLEG): container finished" podID="59fcbe29-f650-41a6-929b-1e25f4e9a80f" containerID="53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f" exitCode=0
Sep 10 17:28:53.207721 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.207722    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"
Sep 10 17:28:53.208165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.207764    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn" event={"ID":"59fcbe29-f650-41a6-929b-1e25f4e9a80f","Type":"ContainerDied","Data":"53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f"}
Sep 10 17:28:53.208165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.207817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn" event={"ID":"59fcbe29-f650-41a6-929b-1e25f4e9a80f","Type":"ContainerDied","Data":"0cb3382c741625db512465f58ea313ca649c162d8b277f669e9bfe5174a46c2f"}
Sep 10 17:28:53.208165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.207842    2556 scope.go:117] "RemoveContainer" containerID="53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f"
Sep 10 17:28:53.217127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.217107    2556 scope.go:117] "RemoveContainer" containerID="53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f"
Sep 10 17:28:53.217411 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:28:53.217387    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f\": container with ID starting with 53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f not found: ID does not exist" containerID="53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f"
Sep 10 17:28:53.217459 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.217420    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f"} err="failed to get container status \"53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f\": rpc error: code = NotFound desc = could not find container \"53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f\": container with ID starting with 53d4b2e543429a2d7661c907025221806b49e4dfaea67bea2b4c573f8077921f not found: ID does not exist"
Sep 10 17:28:53.236313 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.236288    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"]
Sep 10 17:28:53.245759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.245738    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4b53b5-668cff5894-nqxfn"]
Sep 10 17:28:53.272250 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:28:53.272228    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59fcbe29-f650-41a6-929b-1e25f4e9a80f" path="/var/lib/kubelet/pods/59fcbe29-f650-41a6-929b-1e25f4e9a80f/volumes"
Sep 10 17:29:02.434787 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.434745    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"]
Sep 10 17:29:02.435386 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.435368    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fcbe29-f650-41a6-929b-1e25f4e9a80f" containerName="payload-pre-processing"
Sep 10 17:29:02.435434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.435388    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fcbe29-f650-41a6-929b-1e25f4e9a80f" containerName="payload-pre-processing"
Sep 10 17:29:02.435481 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.435471    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fcbe29-f650-41a6-929b-1e25f4e9a80f" containerName="payload-pre-processing"
Sep 10 17:29:02.440211 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.440187    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.444258 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.444235    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6dc6af55\""
Sep 10 17:29:02.467544 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.467516    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0816c7-0b11-4072-8627-a4b75e0622a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.467654 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.467559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcs8q\" (UniqueName: \"kubernetes.io/projected/0a0816c7-0b11-4072-8627-a4b75e0622a3-kube-api-access-hcs8q\") pod \"payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.487297 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.487268    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"]
Sep 10 17:29:02.568688 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.568621    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0816c7-0b11-4072-8627-a4b75e0622a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.568688 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.568697    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcs8q\" (UniqueName: \"kubernetes.io/projected/0a0816c7-0b11-4072-8627-a4b75e0622a3-kube-api-access-hcs8q\") pod \"payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.569368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.569344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0816c7-0b11-4072-8627-a4b75e0622a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.586188 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.586154    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcs8q\" (UniqueName: \"kubernetes.io/projected/0a0816c7-0b11-4072-8627-a4b75e0622a3-kube-api-access-hcs8q\") pod \"payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.752166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.752120    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:02.978623 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.978586    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"]
Sep 10 17:29:02.978833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.978810    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp" podUID="d8fde7a9-8230-4a81-b39e-36c51678dc23" containerName="payload-processing" containerID="cri-o://c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391" gracePeriod=30
Sep 10 17:29:02.994378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.994345    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"]
Sep 10 17:29:02.994610 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:02.994539    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r" podUID="aef743e4-2063-4616-8692-998bb569290a" containerName="payload-pre-processing" containerID="cri-o://daeac91883b482043bfd14123b14808e5504e43f1915ef8b625d2ca60cd1fabe" gracePeriod=30
Sep 10 17:29:03.061129 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.061103    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"]
Sep 10 17:29:03.092807 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:29:03.092777    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0816c7_0b11_4072_8627_a4b75e0622a3.slice/crio-dc40069d9e99c4844bde31e6c328e1b5f595db7f5da54e04d10f3cd2d1e61fc4 WatchSource:0}: Error finding container dc40069d9e99c4844bde31e6c328e1b5f595db7f5da54e04d10f3cd2d1e61fc4: Status 404 returned error can't find the container with id dc40069d9e99c4844bde31e6c328e1b5f595db7f5da54e04d10f3cd2d1e61fc4
Sep 10 17:29:03.262563 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.262532    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:29:03.263966 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.263939    2556 generic.go:356] "Generic (PLEG): container finished" podID="aef743e4-2063-4616-8692-998bb569290a" containerID="daeac91883b482043bfd14123b14808e5504e43f1915ef8b625d2ca60cd1fabe" exitCode=0
Sep 10 17:29:03.264065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.264015    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r" event={"ID":"aef743e4-2063-4616-8692-998bb569290a","Type":"ContainerDied","Data":"daeac91883b482043bfd14123b14808e5504e43f1915ef8b625d2ca60cd1fabe"}
Sep 10 17:29:03.265392 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.265370    2556 generic.go:356] "Generic (PLEG): container finished" podID="d8fde7a9-8230-4a81-b39e-36c51678dc23" containerID="c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391" exitCode=0
Sep 10 17:29:03.265472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.265423    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"
Sep 10 17:29:03.272122 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.272099    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:29:03.274397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274378    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:03.274527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274419    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp" event={"ID":"d8fde7a9-8230-4a81-b39e-36c51678dc23","Type":"ContainerDied","Data":"c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391"}
Sep 10 17:29:03.274527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274441    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp" event={"ID":"d8fde7a9-8230-4a81-b39e-36c51678dc23","Type":"ContainerDied","Data":"ad5b56567c512179c52332d1cac8aa44e87923e827eec56cd710a12bc8227f22"}
Sep 10 17:29:03.274527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274456    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf" event={"ID":"0a0816c7-0b11-4072-8627-a4b75e0622a3","Type":"ContainerStarted","Data":"018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210"}
Sep 10 17:29:03.274527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274469    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf" event={"ID":"0a0816c7-0b11-4072-8627-a4b75e0622a3","Type":"ContainerStarted","Data":"dc40069d9e99c4844bde31e6c328e1b5f595db7f5da54e04d10f3cd2d1e61fc4"}
Sep 10 17:29:03.274527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274490    2556 scope.go:117] "RemoveContainer" containerID="c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391"
Sep 10 17:29:03.274527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274490    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnx4g\" (UniqueName: \"kubernetes.io/projected/d8fde7a9-8230-4a81-b39e-36c51678dc23-kube-api-access-bnx4g\") pod \"d8fde7a9-8230-4a81-b39e-36c51678dc23\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") "
Sep 10 17:29:03.274833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.274585    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fde7a9-8230-4a81-b39e-36c51678dc23-plugins-config-volume\") pod \"d8fde7a9-8230-4a81-b39e-36c51678dc23\" (UID: \"d8fde7a9-8230-4a81-b39e-36c51678dc23\") "
Sep 10 17:29:03.275054 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.275030    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fde7a9-8230-4a81-b39e-36c51678dc23-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8fde7a9-8230-4a81-b39e-36c51678dc23" (UID: "d8fde7a9-8230-4a81-b39e-36c51678dc23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:03.276712 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.276643    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fde7a9-8230-4a81-b39e-36c51678dc23-kube-api-access-bnx4g" (OuterVolumeSpecName: "kube-api-access-bnx4g") pod "d8fde7a9-8230-4a81-b39e-36c51678dc23" (UID: "d8fde7a9-8230-4a81-b39e-36c51678dc23"). InnerVolumeSpecName "kube-api-access-bnx4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:03.284074 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.284055    2556 scope.go:117] "RemoveContainer" containerID="c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391"
Sep 10 17:29:03.284378 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:29:03.284357    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391\": container with ID starting with c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391 not found: ID does not exist" containerID="c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391"
Sep 10 17:29:03.284482 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.284385    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391"} err="failed to get container status \"c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391\": rpc error: code = NotFound desc = could not find container \"c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391\": container with ID starting with c6584ddb9b4044b3ca9b3bd02c94077fb19ab999611eb95b178c816138461391 not found: ID does not exist"
Sep 10 17:29:03.375498 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.375418    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef743e4-2063-4616-8692-998bb569290a-plugins-config-volume\") pod \"aef743e4-2063-4616-8692-998bb569290a\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") "
Sep 10 17:29:03.375643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.375521    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp6bc\" (UniqueName: \"kubernetes.io/projected/aef743e4-2063-4616-8692-998bb569290a-kube-api-access-rp6bc\") pod \"aef743e4-2063-4616-8692-998bb569290a\" (UID: \"aef743e4-2063-4616-8692-998bb569290a\") "
Sep 10 17:29:03.375816 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.375792    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef743e4-2063-4616-8692-998bb569290a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aef743e4-2063-4616-8692-998bb569290a" (UID: "aef743e4-2063-4616-8692-998bb569290a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:03.375882 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.375862    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef743e4-2063-4616-8692-998bb569290a-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:03.375932 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.375890    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fde7a9-8230-4a81-b39e-36c51678dc23-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:03.375932 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.375905    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnx4g\" (UniqueName: \"kubernetes.io/projected/d8fde7a9-8230-4a81-b39e-36c51678dc23-kube-api-access-bnx4g\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:03.377681 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.377637    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef743e4-2063-4616-8692-998bb569290a-kube-api-access-rp6bc" (OuterVolumeSpecName: "kube-api-access-rp6bc") pod "aef743e4-2063-4616-8692-998bb569290a" (UID: "aef743e4-2063-4616-8692-998bb569290a"). InnerVolumeSpecName "kube-api-access-rp6bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:03.388039 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.387978    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf" podStartSLOduration=1.387959894 podStartE2EDuration="1.387959894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:03.386979079 +0000 UTC m=+1028.607103011" watchObservedRunningTime="2026-09-10 17:29:03.387959894 +0000 UTC m=+1028.608083830"
Sep 10 17:29:03.477097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.477061    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6bc\" (UniqueName: \"kubernetes.io/projected/aef743e4-2063-4616-8692-998bb569290a-kube-api-access-rp6bc\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:03.609722 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.609683    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"]
Sep 10 17:29:03.628485 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:03.628396    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-dc5a37-77f66db775-tzrbp"]
Sep 10 17:29:04.273890 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:04.273859    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"
Sep 10 17:29:04.274079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:04.273887    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r" event={"ID":"aef743e4-2063-4616-8692-998bb569290a","Type":"ContainerDied","Data":"4bac657cfdb8e044b2cd641322956b89fe2cf5662513cd8a214320a9565ddf62"}
Sep 10 17:29:04.274079 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:04.273948    2556 scope.go:117] "RemoveContainer" containerID="daeac91883b482043bfd14123b14808e5504e43f1915ef8b625d2ca60cd1fabe"
Sep 10 17:29:04.318687 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:04.318634    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"]
Sep 10 17:29:04.333095 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:04.333071    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-dc5a37-777d975558-9wf6r"]
Sep 10 17:29:05.044727 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.044655    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh" podUID="11b9e423-fd84-4b62-b1ca-7f4a4a826d58" containerName="main" containerID="cri-o://f45d59e636d3049ebc1fe0a9d02bfb7d66d5d7004932edccb387d6c1392b1c09" gracePeriod=45
Sep 10 17:29:05.274210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.274180    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef743e4-2063-4616-8692-998bb569290a" path="/var/lib/kubelet/pods/aef743e4-2063-4616-8692-998bb569290a/volumes"
Sep 10 17:29:05.274584 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.274535    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8fde7a9-8230-4a81-b39e-36c51678dc23" path="/var/lib/kubelet/pods/d8fde7a9-8230-4a81-b39e-36c51678dc23/volumes"
Sep 10 17:29:05.282745 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.282711    2556 generic.go:356] "Generic (PLEG): container finished" podID="11b9e423-fd84-4b62-b1ca-7f4a4a826d58" containerID="f45d59e636d3049ebc1fe0a9d02bfb7d66d5d7004932edccb387d6c1392b1c09" exitCode=0
Sep 10 17:29:05.282887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.282779    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh" event={"ID":"11b9e423-fd84-4b62-b1ca-7f4a4a826d58","Type":"ContainerDied","Data":"f45d59e636d3049ebc1fe0a9d02bfb7d66d5d7004932edccb387d6c1392b1c09"}
Sep 10 17:29:05.282887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.282819    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh" event={"ID":"11b9e423-fd84-4b62-b1ca-7f4a4a826d58","Type":"ContainerDied","Data":"e10419fa047ad73485ce46d6746a0cb587832db9d86ebebfb00600c020933e47"}
Sep 10 17:29:05.282887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.282833    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e10419fa047ad73485ce46d6746a0cb587832db9d86ebebfb00600c020933e47"
Sep 10 17:29:05.293274 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.293249    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:29:05.395037 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.394939    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tls-certs\") pod \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") "
Sep 10 17:29:05.395037 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.394985    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-home\") pod \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") "
Sep 10 17:29:05.395263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395074    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s2xm\" (UniqueName: \"kubernetes.io/projected/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-kube-api-access-5s2xm\") pod \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") "
Sep 10 17:29:05.395263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-model-cache\") pod \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") "
Sep 10 17:29:05.395263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395143    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-dshm\") pod \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") "
Sep 10 17:29:05.395263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395207    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tmp-dir\") pod \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\" (UID: \"11b9e423-fd84-4b62-b1ca-7f4a4a826d58\") "
Sep 10 17:29:05.395470 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395294    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-home" (OuterVolumeSpecName: "home") pod "11b9e423-fd84-4b62-b1ca-7f4a4a826d58" (UID: "11b9e423-fd84-4b62-b1ca-7f4a4a826d58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:05.395470 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395410    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-model-cache" (OuterVolumeSpecName: "model-cache") pod "11b9e423-fd84-4b62-b1ca-7f4a4a826d58" (UID: "11b9e423-fd84-4b62-b1ca-7f4a4a826d58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:05.395577 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395532    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "11b9e423-fd84-4b62-b1ca-7f4a4a826d58" (UID: "11b9e423-fd84-4b62-b1ca-7f4a4a826d58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:05.395626 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395579    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:05.395626 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.395602    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:05.397294 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.397271    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-dshm" (OuterVolumeSpecName: "dshm") pod "11b9e423-fd84-4b62-b1ca-7f4a4a826d58" (UID: "11b9e423-fd84-4b62-b1ca-7f4a4a826d58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:05.397410 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.397322    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "11b9e423-fd84-4b62-b1ca-7f4a4a826d58" (UID: "11b9e423-fd84-4b62-b1ca-7f4a4a826d58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:29:05.397410 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.397327    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-kube-api-access-5s2xm" (OuterVolumeSpecName: "kube-api-access-5s2xm") pod "11b9e423-fd84-4b62-b1ca-7f4a4a826d58" (UID: "11b9e423-fd84-4b62-b1ca-7f4a4a826d58"). InnerVolumeSpecName "kube-api-access-5s2xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:05.496137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.496103    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:05.496137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.496132    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s2xm\" (UniqueName: \"kubernetes.io/projected/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-kube-api-access-5s2xm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:05.496137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.496145    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:05.496405 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:05.496174    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11b9e423-fd84-4b62-b1ca-7f4a4a826d58-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:06.288254 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:06.288221    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"
Sep 10 17:29:06.322570 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:06.322529    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"]
Sep 10 17:29:06.327656 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:06.327622    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-dc5a37/test-model-dc5a37-kserve-6cb4f67995-s7rwh"]
Sep 10 17:29:07.274686 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:07.274633    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11b9e423-fd84-4b62-b1ca-7f4a4a826d58" path="/var/lib/kubelet/pods/11b9e423-fd84-4b62-b1ca-7f4a4a826d58/volumes"
Sep 10 17:29:08.988695 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.988647    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"]
Sep 10 17:29:08.989080 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989065    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef743e4-2063-4616-8692-998bb569290a" containerName="payload-pre-processing"
Sep 10 17:29:08.989121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989082    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef743e4-2063-4616-8692-998bb569290a" containerName="payload-pre-processing"
Sep 10 17:29:08.989121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989105    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8fde7a9-8230-4a81-b39e-36c51678dc23" containerName="payload-processing"
Sep 10 17:29:08.989121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989111    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8fde7a9-8230-4a81-b39e-36c51678dc23" containerName="payload-processing"
Sep 10 17:29:08.989220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989123    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11b9e423-fd84-4b62-b1ca-7f4a4a826d58" containerName="main"
Sep 10 17:29:08.989220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989129    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="11b9e423-fd84-4b62-b1ca-7f4a4a826d58" containerName="main"
Sep 10 17:29:08.989220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989186    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8fde7a9-8230-4a81-b39e-36c51678dc23" containerName="payload-processing"
Sep 10 17:29:08.989220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989193    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="11b9e423-fd84-4b62-b1ca-7f4a4a826d58" containerName="main"
Sep 10 17:29:08.989220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.989199    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef743e4-2063-4616-8692-998bb569290a" containerName="payload-pre-processing"
Sep 10 17:29:08.993068 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.993050    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:08.995744 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.995724    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-eb108f\"/\"kube-root-ca.crt\""
Sep 10 17:29:08.996566 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.996554    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-eb108f\"/\"noten-model-eb108f-kserve-self-signed-certs\""
Sep 10 17:29:08.996731 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.996719    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-eb108f\"/\"default-dockercfg-s4jx6\""
Sep 10 17:29:08.996825 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:08.996802    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-eb108f\"/\"openshift-service-ca.crt\""
Sep 10 17:29:09.006366 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.006341    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"]
Sep 10 17:29:09.026929 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.026896    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-model-cache\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.027071 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.026957    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-home\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.027071 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.026995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8420b148-1ada-4065-9e67-a3f0f0312e46-tls-certs\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.027071 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.027053    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-dshm\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.027208 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.027104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-tmp-dir\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.027208 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.027177    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vq8h\" (UniqueName: \"kubernetes.io/projected/8420b148-1ada-4065-9e67-a3f0f0312e46-kube-api-access-6vq8h\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128371 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128334    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-model-cache\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128552 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-home\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128552 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128410    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8420b148-1ada-4065-9e67-a3f0f0312e46-tls-certs\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128552 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128443    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-dshm\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128552 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128487    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-tmp-dir\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128732 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128567    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vq8h\" (UniqueName: \"kubernetes.io/projected/8420b148-1ada-4065-9e67-a3f0f0312e46-kube-api-access-6vq8h\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128820    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-home\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.128913 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.128873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-model-cache\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.129105 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.129079    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-tmp-dir\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.130864 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.130837    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-dshm\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.131247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.131227    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8420b148-1ada-4065-9e67-a3f0f0312e46-tls-certs\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.147432 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.147406    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vq8h\" (UniqueName: \"kubernetes.io/projected/8420b148-1ada-4065-9e67-a3f0f0312e46-kube-api-access-6vq8h\") pod \"noten-model-eb108f-kserve-5569b749cf-hwssh\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") " pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.302003 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.301913    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:09.588988 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:09.588960    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"]
Sep 10 17:29:09.590917 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:29:09.590888    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8420b148_1ada_4065_9e67_a3f0f0312e46.slice/crio-ceb29dea0be3b6a24dc44238963886329823e42e8ebf455f855737842b16b5b0 WatchSource:0}: Error finding container ceb29dea0be3b6a24dc44238963886329823e42e8ebf455f855737842b16b5b0: Status 404 returned error can't find the container with id ceb29dea0be3b6a24dc44238963886329823e42e8ebf455f855737842b16b5b0
Sep 10 17:29:10.311904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:10.311862    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh" event={"ID":"8420b148-1ada-4065-9e67-a3f0f0312e46","Type":"ContainerStarted","Data":"3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3"}
Sep 10 17:29:10.312328 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:10.311915    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh" event={"ID":"8420b148-1ada-4065-9e67-a3f0f0312e46","Type":"ContainerStarted","Data":"ceb29dea0be3b6a24dc44238963886329823e42e8ebf455f855737842b16b5b0"}
Sep 10 17:29:10.341769 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:10.341719    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh" podStartSLOduration=2.341703919 podStartE2EDuration="2.341703919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:10.33898136 +0000 UTC m=+1035.559105293" watchObservedRunningTime="2026-09-10 17:29:10.341703919 +0000 UTC m=+1035.561827852"
Sep 10 17:29:12.231055 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.231014    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"]
Sep 10 17:29:12.234617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.234599    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:12.240091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.240061    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:29:12.240248 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.240130    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:29:12.253037 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.253010    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"]
Sep 10 17:29:12.257206 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.257176    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9f5890-6c25-419d-ab65-9a8001e462ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9\" (UID: \"8f9f5890-6c25-419d-ab65-9a8001e462ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:12.358407 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.358372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9f5890-6c25-419d-ab65-9a8001e462ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9\" (UID: \"8f9f5890-6c25-419d-ab65-9a8001e462ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:12.359280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.359247    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9f5890-6c25-419d-ab65-9a8001e462ce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9\" (UID: \"8f9f5890-6c25-419d-ab65-9a8001e462ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:12.546196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.546106    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:12.820833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.820748    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"]
Sep 10 17:29:12.823272 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:29:12.823244    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9f5890_6c25_419d_ab65_9a8001e462ce.slice/crio-fe0729450a5b963d2e215a1e5c9a9187cca61928a66548ff194cc2815135bf26 WatchSource:0}: Error finding container fe0729450a5b963d2e215a1e5c9a9187cca61928a66548ff194cc2815135bf26: Status 404 returned error can't find the container with id fe0729450a5b963d2e215a1e5c9a9187cca61928a66548ff194cc2815135bf26
Sep 10 17:29:12.825590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:12.825574    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:29:13.329183 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:13.329144    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9" event={"ID":"8f9f5890-6c25-419d-ab65-9a8001e462ce","Type":"ContainerStarted","Data":"fe0729450a5b963d2e215a1e5c9a9187cca61928a66548ff194cc2815135bf26"}
Sep 10 17:29:14.277524 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:14.277492    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:18.364740 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:18.364696    2556 generic.go:356] "Generic (PLEG): container finished" podID="8f9f5890-6c25-419d-ab65-9a8001e462ce" containerID="655f225faf1753d959e331237616e157ef5e24f82db6a6cd47ca0584201ed12c" exitCode=0
Sep 10 17:29:18.365120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:18.364751    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9" event={"ID":"8f9f5890-6c25-419d-ab65-9a8001e462ce","Type":"ContainerDied","Data":"655f225faf1753d959e331237616e157ef5e24f82db6a6cd47ca0584201ed12c"}
Sep 10 17:29:19.302357 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.302319    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:19.302357 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.302366    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:19.314503 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.314478    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:19.381549 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.381517    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:19.517676 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.517638    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:19.629104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.629011    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9f5890-6c25-419d-ab65-9a8001e462ce-maas-api-service-ca\") pod \"8f9f5890-6c25-419d-ab65-9a8001e462ce\" (UID: \"8f9f5890-6c25-419d-ab65-9a8001e462ce\") "
Sep 10 17:29:19.629434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.629410    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9f5890-6c25-419d-ab65-9a8001e462ce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f9f5890-6c25-419d-ab65-9a8001e462ce" (UID: "8f9f5890-6c25-419d-ab65-9a8001e462ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:19.730016 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:19.729976    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f9f5890-6c25-419d-ab65-9a8001e462ce-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:20.376073 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:20.376038    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"
Sep 10 17:29:20.376282 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:20.376079    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9" event={"ID":"8f9f5890-6c25-419d-ab65-9a8001e462ce","Type":"ContainerDied","Data":"fe0729450a5b963d2e215a1e5c9a9187cca61928a66548ff194cc2815135bf26"}
Sep 10 17:29:20.376282 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:20.376133    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe0729450a5b963d2e215a1e5c9a9187cca61928a66548ff194cc2815135bf26"
Sep 10 17:29:23.543172 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:23.543138    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"]
Sep 10 17:29:23.845765 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:23.845648    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"]
Sep 10 17:29:23.846001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:23.845969    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb" podUID="6b269482-1d79-4c96-9835-afd17f10bfdb" containerName="payload-processing" containerID="cri-o://3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165" gracePeriod=30
Sep 10 17:29:23.857269 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:23.857235    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"]
Sep 10 17:29:23.857510 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:23.857471    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5" podUID="4caf9198-044f-407a-aa04-770d7f858b7b" containerName="payload-pre-processing" containerID="cri-o://1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927" gracePeriod=30
Sep 10 17:29:24.062115 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.062064    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"]
Sep 10 17:29:24.068593 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.068565    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-75bd9a-vb4f9"]
Sep 10 17:29:24.107741 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.107679    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:29:24.140012 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.139978    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:29:24.174931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.174902    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b269482-1d79-4c96-9835-afd17f10bfdb-plugins-config-volume\") pod \"6b269482-1d79-4c96-9835-afd17f10bfdb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") "
Sep 10 17:29:24.175118 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.175009    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sz7w\" (UniqueName: \"kubernetes.io/projected/6b269482-1d79-4c96-9835-afd17f10bfdb-kube-api-access-6sz7w\") pod \"6b269482-1d79-4c96-9835-afd17f10bfdb\" (UID: \"6b269482-1d79-4c96-9835-afd17f10bfdb\") "
Sep 10 17:29:24.175280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.175256    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b269482-1d79-4c96-9835-afd17f10bfdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b269482-1d79-4c96-9835-afd17f10bfdb" (UID: "6b269482-1d79-4c96-9835-afd17f10bfdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:24.175364 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.175349    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b269482-1d79-4c96-9835-afd17f10bfdb-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:24.177065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.177036    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b269482-1d79-4c96-9835-afd17f10bfdb-kube-api-access-6sz7w" (OuterVolumeSpecName: "kube-api-access-6sz7w") pod "6b269482-1d79-4c96-9835-afd17f10bfdb" (UID: "6b269482-1d79-4c96-9835-afd17f10bfdb"). InnerVolumeSpecName "kube-api-access-6sz7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:24.276291 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.276255    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caf9198-044f-407a-aa04-770d7f858b7b-plugins-config-volume\") pod \"4caf9198-044f-407a-aa04-770d7f858b7b\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") "
Sep 10 17:29:24.276485 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.276345    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ztvg\" (UniqueName: \"kubernetes.io/projected/4caf9198-044f-407a-aa04-770d7f858b7b-kube-api-access-6ztvg\") pod \"4caf9198-044f-407a-aa04-770d7f858b7b\" (UID: \"4caf9198-044f-407a-aa04-770d7f858b7b\") "
Sep 10 17:29:24.276685 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.276633    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4caf9198-044f-407a-aa04-770d7f858b7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4caf9198-044f-407a-aa04-770d7f858b7b" (UID: "4caf9198-044f-407a-aa04-770d7f858b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:24.276766 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.276745    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sz7w\" (UniqueName: \"kubernetes.io/projected/6b269482-1d79-4c96-9835-afd17f10bfdb-kube-api-access-6sz7w\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:24.276804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.276770    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4caf9198-044f-407a-aa04-770d7f858b7b-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:24.278425 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.278404    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caf9198-044f-407a-aa04-770d7f858b7b-kube-api-access-6ztvg" (OuterVolumeSpecName: "kube-api-access-6ztvg") pod "4caf9198-044f-407a-aa04-770d7f858b7b" (UID: "4caf9198-044f-407a-aa04-770d7f858b7b"). InnerVolumeSpecName "kube-api-access-6ztvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:24.378001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.377896    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztvg\" (UniqueName: \"kubernetes.io/projected/4caf9198-044f-407a-aa04-770d7f858b7b-kube-api-access-6ztvg\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:24.397791 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.397755    2556 generic.go:356] "Generic (PLEG): container finished" podID="4caf9198-044f-407a-aa04-770d7f858b7b" containerID="1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927" exitCode=0
Sep 10 17:29:24.397999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.397808    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"
Sep 10 17:29:24.397999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.397841    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5" event={"ID":"4caf9198-044f-407a-aa04-770d7f858b7b","Type":"ContainerDied","Data":"1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927"}
Sep 10 17:29:24.397999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.397893    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5" event={"ID":"4caf9198-044f-407a-aa04-770d7f858b7b","Type":"ContainerDied","Data":"8397dec901ce23db363944f3832bfc2a9f7c6c6762e980ed773e83cafd41e800"}
Sep 10 17:29:24.397999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.397921    2556 scope.go:117] "RemoveContainer" containerID="1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927"
Sep 10 17:29:24.399231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.399211    2556 generic.go:356] "Generic (PLEG): container finished" podID="6b269482-1d79-4c96-9835-afd17f10bfdb" containerID="3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165" exitCode=0
Sep 10 17:29:24.399378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.399250    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb" event={"ID":"6b269482-1d79-4c96-9835-afd17f10bfdb","Type":"ContainerDied","Data":"3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165"}
Sep 10 17:29:24.399378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.399259    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"
Sep 10 17:29:24.399378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.399285    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb" event={"ID":"6b269482-1d79-4c96-9835-afd17f10bfdb","Type":"ContainerDied","Data":"03d42a13f7bb319e66be8c8969660f9b6b693fafac96e06af26e94d62382177b"}
Sep 10 17:29:24.409353 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.409332    2556 scope.go:117] "RemoveContainer" containerID="1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927"
Sep 10 17:29:24.409648 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:29:24.409620    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927\": container with ID starting with 1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927 not found: ID does not exist" containerID="1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927"
Sep 10 17:29:24.409756 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.409648    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927"} err="failed to get container status \"1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927\": rpc error: code = NotFound desc = could not find container \"1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927\": container with ID starting with 1ae4daf786bd0f964eeb120c434cc32a1444ce18be90b3ab79c3664e91586927 not found: ID does not exist"
Sep 10 17:29:24.409756 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.409687    2556 scope.go:117] "RemoveContainer" containerID="3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165"
Sep 10 17:29:24.418292 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.418270    2556 scope.go:117] "RemoveContainer" containerID="3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165"
Sep 10 17:29:24.418555 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:29:24.418536    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165\": container with ID starting with 3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165 not found: ID does not exist" containerID="3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165"
Sep 10 17:29:24.418609 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.418563    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165"} err="failed to get container status \"3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165\": rpc error: code = NotFound desc = could not find container \"3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165\": container with ID starting with 3bcdbc23ba337f31bf170928af692a4deb80245c22ddb78542dec965939d5165 not found: ID does not exist"
Sep 10 17:29:24.423091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.423069    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"]
Sep 10 17:29:24.427574 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.427551    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-75bd9a-6dc4c5874-788b5"]
Sep 10 17:29:24.440364 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.440273    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"]
Sep 10 17:29:24.442368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:24.442348    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-75bd9a-7d846bc4c5-ps7sb"]
Sep 10 17:29:25.271563 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:25.271526    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4caf9198-044f-407a-aa04-770d7f858b7b" path="/var/lib/kubelet/pods/4caf9198-044f-407a-aa04-770d7f858b7b/volumes"
Sep 10 17:29:25.271959 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:25.271918    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b269482-1d79-4c96-9835-afd17f10bfdb" path="/var/lib/kubelet/pods/6b269482-1d79-4c96-9835-afd17f10bfdb/volumes"
Sep 10 17:29:25.272212 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:25.272197    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9f5890-6c25-419d-ab65-9a8001e462ce" path="/var/lib/kubelet/pods/8f9f5890-6c25-419d-ab65-9a8001e462ce/volumes"
Sep 10 17:29:34.490160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490122    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"]
Sep 10 17:29:34.490652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490560    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caf9198-044f-407a-aa04-770d7f858b7b" containerName="payload-pre-processing"
Sep 10 17:29:34.490652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490572    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caf9198-044f-407a-aa04-770d7f858b7b" containerName="payload-pre-processing"
Sep 10 17:29:34.490652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490592    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9f5890-6c25-419d-ab65-9a8001e462ce" containerName="revoke-keys"
Sep 10 17:29:34.490652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490597    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9f5890-6c25-419d-ab65-9a8001e462ce" containerName="revoke-keys"
Sep 10 17:29:34.490652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490610    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b269482-1d79-4c96-9835-afd17f10bfdb" containerName="payload-processing"
Sep 10 17:29:34.490652 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490616    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b269482-1d79-4c96-9835-afd17f10bfdb" containerName="payload-processing"
Sep 10 17:29:34.491019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490713    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caf9198-044f-407a-aa04-770d7f858b7b" containerName="payload-pre-processing"
Sep 10 17:29:34.491019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490722    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b269482-1d79-4c96-9835-afd17f10bfdb" containerName="payload-processing"
Sep 10 17:29:34.491019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.490730    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9f5890-6c25-419d-ab65-9a8001e462ce" containerName="revoke-keys"
Sep 10 17:29:34.494003 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.493973    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:34.496974 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.496951    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:29:34.497100 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.497043    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:29:34.505397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.505372    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"]
Sep 10 17:29:34.583275 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.583178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1019ca97-9ba9-4c7d-ae14-f2f8bb518335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss\" (UID: \"1019ca97-9ba9-4c7d-ae14-f2f8bb518335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:34.684728 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.684683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1019ca97-9ba9-4c7d-ae14-f2f8bb518335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss\" (UID: \"1019ca97-9ba9-4c7d-ae14-f2f8bb518335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:34.685265 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.685242    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1019ca97-9ba9-4c7d-ae14-f2f8bb518335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss\" (UID: \"1019ca97-9ba9-4c7d-ae14-f2f8bb518335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:34.807059 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:34.807023    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:35.097853 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:35.097804    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"]
Sep 10 17:29:35.103238 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:29:35.103197    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1019ca97_9ba9_4c7d_ae14_f2f8bb518335.slice/crio-e88505d05af8a77a4ac7b668f04221ffcce97c632554af5944ff689ff3b7aa1d WatchSource:0}: Error finding container e88505d05af8a77a4ac7b668f04221ffcce97c632554af5944ff689ff3b7aa1d: Status 404 returned error can't find the container with id e88505d05af8a77a4ac7b668f04221ffcce97c632554af5944ff689ff3b7aa1d
Sep 10 17:29:35.458680 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:35.458628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss" event={"ID":"1019ca97-9ba9-4c7d-ae14-f2f8bb518335","Type":"ContainerStarted","Data":"e88505d05af8a77a4ac7b668f04221ffcce97c632554af5944ff689ff3b7aa1d"}
Sep 10 17:29:36.467419 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:36.467382    2556 generic.go:356] "Generic (PLEG): container finished" podID="1019ca97-9ba9-4c7d-ae14-f2f8bb518335" containerID="958094f8c08228e54b99e66d2648fa0573e55d2e593688eed834af77d2c9aaca" exitCode=0
Sep 10 17:29:36.467883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:36.467463    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss" event={"ID":"1019ca97-9ba9-4c7d-ae14-f2f8bb518335","Type":"ContainerDied","Data":"958094f8c08228e54b99e66d2648fa0573e55d2e593688eed834af77d2c9aaca"}
Sep 10 17:29:37.671365 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:37.671337    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:37.820535 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:37.820446    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1019ca97-9ba9-4c7d-ae14-f2f8bb518335-maas-api-service-ca\") pod \"1019ca97-9ba9-4c7d-ae14-f2f8bb518335\" (UID: \"1019ca97-9ba9-4c7d-ae14-f2f8bb518335\") "
Sep 10 17:29:37.820894 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:37.820873    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1019ca97-9ba9-4c7d-ae14-f2f8bb518335-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1019ca97-9ba9-4c7d-ae14-f2f8bb518335" (UID: "1019ca97-9ba9-4c7d-ae14-f2f8bb518335"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:37.922426 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:37.922387    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1019ca97-9ba9-4c7d-ae14-f2f8bb518335-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:38.480780 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.480747    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"
Sep 10 17:29:38.480780 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.480763    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss" event={"ID":"1019ca97-9ba9-4c7d-ae14-f2f8bb518335","Type":"ContainerDied","Data":"e88505d05af8a77a4ac7b668f04221ffcce97c632554af5944ff689ff3b7aa1d"}
Sep 10 17:29:38.480994 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.480812    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88505d05af8a77a4ac7b668f04221ffcce97c632554af5944ff689ff3b7aa1d"
Sep 10 17:29:38.569178 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.569137    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh" podUID="8420b148-1ada-4065-9e67-a3f0f0312e46" containerName="main" containerID="cri-o://3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3" gracePeriod=45
Sep 10 17:29:38.619577 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:29:38.619541    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8420b148_1ada_4065_9e67_a3f0f0312e46.slice/crio-3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 17:29:38.882336 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.882303    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:38.900554 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.900521    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"]
Sep 10 17:29:38.901043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.901024    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1019ca97-9ba9-4c7d-ae14-f2f8bb518335" containerName="revoke-keys"
Sep 10 17:29:38.901043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.901044    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1019ca97-9ba9-4c7d-ae14-f2f8bb518335" containerName="revoke-keys"
Sep 10 17:29:38.901177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.901067    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8420b148-1ada-4065-9e67-a3f0f0312e46" containerName="main"
Sep 10 17:29:38.901177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.901073    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8420b148-1ada-4065-9e67-a3f0f0312e46" containerName="main"
Sep 10 17:29:38.901177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.901143    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1019ca97-9ba9-4c7d-ae14-f2f8bb518335" containerName="revoke-keys"
Sep 10 17:29:38.901177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.901152    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="8420b148-1ada-4065-9e67-a3f0f0312e46" containerName="main"
Sep 10 17:29:38.904202 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.904182    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:38.910704 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.910679    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-be583c-dockercfg-fcslf\""
Sep 10 17:29:38.911602 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.911486    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-be583c\""
Sep 10 17:29:38.941022 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.940907    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"]
Sep 10 17:29:38.941186 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941064    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-home\") pod \"8420b148-1ada-4065-9e67-a3f0f0312e46\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") "
Sep 10 17:29:38.941186 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941133    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-tmp-dir\") pod \"8420b148-1ada-4065-9e67-a3f0f0312e46\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") "
Sep 10 17:29:38.941306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941191    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-dshm\") pod \"8420b148-1ada-4065-9e67-a3f0f0312e46\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") "
Sep 10 17:29:38.941306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941231    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8420b148-1ada-4065-9e67-a3f0f0312e46-tls-certs\") pod \"8420b148-1ada-4065-9e67-a3f0f0312e46\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") "
Sep 10 17:29:38.941306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941268    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vq8h\" (UniqueName: \"kubernetes.io/projected/8420b148-1ada-4065-9e67-a3f0f0312e46-kube-api-access-6vq8h\") pod \"8420b148-1ada-4065-9e67-a3f0f0312e46\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") "
Sep 10 17:29:38.941455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941319    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-model-cache\") pod \"8420b148-1ada-4065-9e67-a3f0f0312e46\" (UID: \"8420b148-1ada-4065-9e67-a3f0f0312e46\") "
Sep 10 17:29:38.941455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941319    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-home" (OuterVolumeSpecName: "home") pod "8420b148-1ada-4065-9e67-a3f0f0312e46" (UID: "8420b148-1ada-4065-9e67-a3f0f0312e46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:38.941455 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941400    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8420b148-1ada-4065-9e67-a3f0f0312e46" (UID: "8420b148-1ada-4065-9e67-a3f0f0312e46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:38.941604 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5c6p\" (UniqueName: \"kubernetes.io/projected/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-kube-api-access-x5c6p\") pod \"payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:38.941604 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941560    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:38.941765 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941750    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:38.941822 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941766    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:38.941871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.941817    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-model-cache" (OuterVolumeSpecName: "model-cache") pod "8420b148-1ada-4065-9e67-a3f0f0312e46" (UID: "8420b148-1ada-4065-9e67-a3f0f0312e46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:38.944565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.944078    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-dshm" (OuterVolumeSpecName: "dshm") pod "8420b148-1ada-4065-9e67-a3f0f0312e46" (UID: "8420b148-1ada-4065-9e67-a3f0f0312e46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:29:38.944982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.944957    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8420b148-1ada-4065-9e67-a3f0f0312e46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8420b148-1ada-4065-9e67-a3f0f0312e46" (UID: "8420b148-1ada-4065-9e67-a3f0f0312e46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:29:38.945096 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:38.945066    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8420b148-1ada-4065-9e67-a3f0f0312e46-kube-api-access-6vq8h" (OuterVolumeSpecName: "kube-api-access-6vq8h") pod "8420b148-1ada-4065-9e67-a3f0f0312e46" (UID: "8420b148-1ada-4065-9e67-a3f0f0312e46"). InnerVolumeSpecName "kube-api-access-6vq8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:39.003682 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.003389    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"]
Sep 10 17:29:39.008590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.008563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.015404 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.015344    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"]
Sep 10 17:29:39.042525 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5c6p\" (UniqueName: \"kubernetes.io/projected/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-kube-api-access-x5c6p\") pod \"payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:39.042525 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042528    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:39.042747 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042593    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4txr\" (UniqueName: \"kubernetes.io/projected/cadaf1a8-8e25-4536-b652-a61a0a16afb3-kube-api-access-x4txr\") pod \"payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.042747 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042626    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadaf1a8-8e25-4536-b652-a61a0a16afb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.042839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042785    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:39.042839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042809    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8420b148-1ada-4065-9e67-a3f0f0312e46-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:39.042839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042825    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vq8h\" (UniqueName: \"kubernetes.io/projected/8420b148-1ada-4065-9e67-a3f0f0312e46-kube-api-access-6vq8h\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:39.042959 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.042839    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8420b148-1ada-4065-9e67-a3f0f0312e46-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:39.043437 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.043399    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:39.073190 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.073095    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5c6p\" (UniqueName: \"kubernetes.io/projected/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-kube-api-access-x5c6p\") pod \"payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:39.143405 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.143350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4txr\" (UniqueName: \"kubernetes.io/projected/cadaf1a8-8e25-4536-b652-a61a0a16afb3-kube-api-access-x4txr\") pod \"payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.143405 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.143410    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadaf1a8-8e25-4536-b652-a61a0a16afb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.144160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.144132    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadaf1a8-8e25-4536-b652-a61a0a16afb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.154422 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.154392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4txr\" (UniqueName: \"kubernetes.io/projected/cadaf1a8-8e25-4536-b652-a61a0a16afb3-kube-api-access-x4txr\") pod \"payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.216191 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.216151    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:39.324759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.321823    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:39.488443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.488386    2556 generic.go:356] "Generic (PLEG): container finished" podID="8420b148-1ada-4065-9e67-a3f0f0312e46" containerID="3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3" exitCode=0
Sep 10 17:29:39.488443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.488434    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"
Sep 10 17:29:39.488683 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.488485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh" event={"ID":"8420b148-1ada-4065-9e67-a3f0f0312e46","Type":"ContainerDied","Data":"3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3"}
Sep 10 17:29:39.488683 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.488543    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh" event={"ID":"8420b148-1ada-4065-9e67-a3f0f0312e46","Type":"ContainerDied","Data":"ceb29dea0be3b6a24dc44238963886329823e42e8ebf455f855737842b16b5b0"}
Sep 10 17:29:39.488683 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.488570    2556 scope.go:117] "RemoveContainer" containerID="3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3"
Sep 10 17:29:39.497823 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.497799    2556 scope.go:117] "RemoveContainer" containerID="3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3"
Sep 10 17:29:39.498160 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:29:39.498132    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3\": container with ID starting with 3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3 not found: ID does not exist" containerID="3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3"
Sep 10 17:29:39.498211 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.498168    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3"} err="failed to get container status \"3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3\": rpc error: code = NotFound desc = could not find container \"3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3\": container with ID starting with 3bcc37d6a8212af422a95de34c922b6d5a2e2397d65814916ca397ca824542e3 not found: ID does not exist"
Sep 10 17:29:39.517808 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.517774    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"]
Sep 10 17:29:39.523357 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.523323    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"]
Sep 10 17:29:39.525617 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:29:39.525591    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f6e8fc_1237_43ef_a536_b3dd7aa5b362.slice/crio-88fa5f2aac28766107f3a80c3ce50cea20fa34804afb826779090932ed7fb23f WatchSource:0}: Error finding container 88fa5f2aac28766107f3a80c3ce50cea20fa34804afb826779090932ed7fb23f: Status 404 returned error can't find the container with id 88fa5f2aac28766107f3a80c3ce50cea20fa34804afb826779090932ed7fb23f
Sep 10 17:29:39.529832 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.529809    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-eb108f/noten-model-eb108f-kserve-5569b749cf-hwssh"]
Sep 10 17:29:39.629094 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:39.629064    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"]
Sep 10 17:29:39.631993 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:29:39.631963    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadaf1a8_8e25_4536_b652_a61a0a16afb3.slice/crio-643678f943d743486f893a521d3024ffddab4d3abf3023315413b173e01f94ed WatchSource:0}: Error finding container 643678f943d743486f893a521d3024ffddab4d3abf3023315413b173e01f94ed: Status 404 returned error can't find the container with id 643678f943d743486f893a521d3024ffddab4d3abf3023315413b173e01f94ed
Sep 10 17:29:40.496100 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.496057    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws" event={"ID":"32f6e8fc-1237-43ef-a536-b3dd7aa5b362","Type":"ContainerStarted","Data":"5e1741c5cae1824d8b19c4f2691eb7dcf56d90efd8dcb6ca65b93df7815e342e"}
Sep 10 17:29:40.496567 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.496129    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws" event={"ID":"32f6e8fc-1237-43ef-a536-b3dd7aa5b362","Type":"ContainerStarted","Data":"88fa5f2aac28766107f3a80c3ce50cea20fa34804afb826779090932ed7fb23f"}
Sep 10 17:29:40.496567 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.496164    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:29:40.497821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.497793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4" event={"ID":"cadaf1a8-8e25-4536-b652-a61a0a16afb3","Type":"ContainerStarted","Data":"0623d4d7c1a39e56a1e727b9fda337b07d709243483b019b8bb33dbf52d0d216"}
Sep 10 17:29:40.497949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.497847    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:40.497949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.497873    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4" event={"ID":"cadaf1a8-8e25-4536-b652-a61a0a16afb3","Type":"ContainerStarted","Data":"643678f943d743486f893a521d3024ffddab4d3abf3023315413b173e01f94ed"}
Sep 10 17:29:40.519742 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.519690    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws" podStartSLOduration=2.519672351 podStartE2EDuration="2.519672351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:40.518337661 +0000 UTC m=+1065.738461610" watchObservedRunningTime="2026-09-10 17:29:40.519672351 +0000 UTC m=+1065.739796275"
Sep 10 17:29:40.552263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:40.552212    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4" podStartSLOduration=2.55219809 podStartE2EDuration="2.55219809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:29:40.550328107 +0000 UTC m=+1065.770452042" watchObservedRunningTime="2026-09-10 17:29:40.55219809 +0000 UTC m=+1065.772322118"
Sep 10 17:29:41.271738 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:41.271702    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8420b148-1ada-4065-9e67-a3f0f0312e46" path="/var/lib/kubelet/pods/8420b148-1ada-4065-9e67-a3f0f0312e46/volumes"
Sep 10 17:29:45.641425 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:45.641386    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"]
Sep 10 17:29:45.641824 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:45.641643    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf" podUID="0a0816c7-0b11-4072-8627-a4b75e0622a3" containerName="payload-pre-processing" containerID="cri-o://018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210" gracePeriod=30
Sep 10 17:29:45.901466 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:45.901397    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:45.963976 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:45.963941    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"]
Sep 10 17:29:45.972643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:45.972611    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6dc6af55-4vsss"]
Sep 10 17:29:46.011871 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.011835    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcs8q\" (UniqueName: \"kubernetes.io/projected/0a0816c7-0b11-4072-8627-a4b75e0622a3-kube-api-access-hcs8q\") pod \"0a0816c7-0b11-4072-8627-a4b75e0622a3\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") "
Sep 10 17:29:46.012053 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.011944    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0816c7-0b11-4072-8627-a4b75e0622a3-plugins-config-volume\") pod \"0a0816c7-0b11-4072-8627-a4b75e0622a3\" (UID: \"0a0816c7-0b11-4072-8627-a4b75e0622a3\") "
Sep 10 17:29:46.012321 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.012295    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0816c7-0b11-4072-8627-a4b75e0622a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a0816c7-0b11-4072-8627-a4b75e0622a3" (UID: "0a0816c7-0b11-4072-8627-a4b75e0622a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:29:46.014087 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.014061    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0816c7-0b11-4072-8627-a4b75e0622a3-kube-api-access-hcs8q" (OuterVolumeSpecName: "kube-api-access-hcs8q") pod "0a0816c7-0b11-4072-8627-a4b75e0622a3" (UID: "0a0816c7-0b11-4072-8627-a4b75e0622a3"). InnerVolumeSpecName "kube-api-access-hcs8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:29:46.113144 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.113100    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcs8q\" (UniqueName: \"kubernetes.io/projected/0a0816c7-0b11-4072-8627-a4b75e0622a3-kube-api-access-hcs8q\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:46.113144 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.113134    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a0816c7-0b11-4072-8627-a4b75e0622a3-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:29:46.532299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.532264    2556 generic.go:356] "Generic (PLEG): container finished" podID="0a0816c7-0b11-4072-8627-a4b75e0622a3" containerID="018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210" exitCode=0
Sep 10 17:29:46.532519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.532316    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"
Sep 10 17:29:46.532519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.532360    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf" event={"ID":"0a0816c7-0b11-4072-8627-a4b75e0622a3","Type":"ContainerDied","Data":"018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210"}
Sep 10 17:29:46.532519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.532418    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf" event={"ID":"0a0816c7-0b11-4072-8627-a4b75e0622a3","Type":"ContainerDied","Data":"dc40069d9e99c4844bde31e6c328e1b5f595db7f5da54e04d10f3cd2d1e61fc4"}
Sep 10 17:29:46.532519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.532448    2556 scope.go:117] "RemoveContainer" containerID="018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210"
Sep 10 17:29:46.545502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.545477    2556 scope.go:117] "RemoveContainer" containerID="018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210"
Sep 10 17:29:46.545897 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:29:46.545871    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210\": container with ID starting with 018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210 not found: ID does not exist" containerID="018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210"
Sep 10 17:29:46.546005 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.545906    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210"} err="failed to get container status \"018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210\": rpc error: code = NotFound desc = could not find container \"018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210\": container with ID starting with 018b5cd2238a9e84b7fd10998ac53e779234b036367857fb849b2f51ce6f0210 not found: ID does not exist"
Sep 10 17:29:46.561803 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.561776    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"]
Sep 10 17:29:46.571370 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:46.571337    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6dc6af55-6768f4fc5d-j6xnf"]
Sep 10 17:29:47.271591 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:47.271512    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a0816c7-0b11-4072-8627-a4b75e0622a3" path="/var/lib/kubelet/pods/0a0816c7-0b11-4072-8627-a4b75e0622a3/volumes"
Sep 10 17:29:47.271976 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:47.271941    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1019ca97-9ba9-4c7d-ae14-f2f8bb518335" path="/var/lib/kubelet/pods/1019ca97-9ba9-4c7d-ae14-f2f8bb518335/volumes"
Sep 10 17:29:51.506445 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:51.506413    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:29:51.506950 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:29:51.506826    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:30:00.172295 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.172258    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"]
Sep 10 17:30:00.172740 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.172705    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a0816c7-0b11-4072-8627-a4b75e0622a3" containerName="payload-pre-processing"
Sep 10 17:30:00.172740 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.172717    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a0816c7-0b11-4072-8627-a4b75e0622a3" containerName="payload-pre-processing"
Sep 10 17:30:00.172817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.172785    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a0816c7-0b11-4072-8627-a4b75e0622a3" containerName="payload-pre-processing"
Sep 10 17:30:00.177369 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.177342    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"
Sep 10 17:30:00.180404 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.180376    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:30:00.188077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.188047    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"]
Sep 10 17:30:00.189115 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.189084    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"
Sep 10 17:30:00.465031 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.465001    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"]
Sep 10 17:30:00.466290 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:30:00.466264    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db1792c_8a30_4ed8_9ee8_806ed1238596.slice/crio-82b568012988d8d19a801bcf2f2e9522638e5ac6aefdf287b638f666ca32973c WatchSource:0}: Error finding container 82b568012988d8d19a801bcf2f2e9522638e5ac6aefdf287b638f666ca32973c: Status 404 returned error can't find the container with id 82b568012988d8d19a801bcf2f2e9522638e5ac6aefdf287b638f666ca32973c
Sep 10 17:30:00.614419 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:00.614379    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq" event={"ID":"3db1792c-8a30-4ed8-9ee8-806ed1238596","Type":"ContainerStarted","Data":"82b568012988d8d19a801bcf2f2e9522638e5ac6aefdf287b638f666ca32973c"}
Sep 10 17:30:01.621217 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:01.621106    2556 generic.go:356] "Generic (PLEG): container finished" podID="3db1792c-8a30-4ed8-9ee8-806ed1238596" containerID="6f48c92e2cd22c141138ad0815c91fcecc487d05c6bb1ad060b14a3135527351" exitCode=0
Sep 10 17:30:01.621217 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:01.621167    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq" event={"ID":"3db1792c-8a30-4ed8-9ee8-806ed1238596","Type":"ContainerDied","Data":"6f48c92e2cd22c141138ad0815c91fcecc487d05c6bb1ad060b14a3135527351"}
Sep 10 17:30:02.764768 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:02.764744    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"
Sep 10 17:30:03.633010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:03.632933    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"
Sep 10 17:30:03.633159 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:03.632932    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq" event={"ID":"3db1792c-8a30-4ed8-9ee8-806ed1238596","Type":"ContainerDied","Data":"82b568012988d8d19a801bcf2f2e9522638e5ac6aefdf287b638f666ca32973c"}
Sep 10 17:30:03.633159 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:03.633057    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82b568012988d8d19a801bcf2f2e9522638e5ac6aefdf287b638f666ca32973c"
Sep 10 17:30:16.469182 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.469151    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"]
Sep 10 17:30:16.469584 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.469571    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3db1792c-8a30-4ed8-9ee8-806ed1238596" containerName="cleanup"
Sep 10 17:30:16.469629 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.469586    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3db1792c-8a30-4ed8-9ee8-806ed1238596" containerName="cleanup"
Sep 10 17:30:16.469679 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.469650    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3db1792c-8a30-4ed8-9ee8-806ed1238596" containerName="cleanup"
Sep 10 17:30:16.473197 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.473179    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.476238 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.476218    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-33b7bb\""
Sep 10 17:30:16.476238 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.476227    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-33b7bb-dockercfg-b644m\""
Sep 10 17:30:16.489200 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.489173    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"]
Sep 10 17:30:16.517367 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.517339    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z77jj\" (UniqueName: \"kubernetes.io/projected/956b4ddd-1483-402a-9b72-47871d24e448-kube-api-access-z77jj\") pod \"payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.517512 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.517467    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b4ddd-1483-402a-9b72-47871d24e448-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.560639 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.560604    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"]
Sep 10 17:30:16.564520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.564498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.585844 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.585817    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"]
Sep 10 17:30:16.618831 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.618795    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b4ddd-1483-402a-9b72-47871d24e448-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.618997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.618837    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.618997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.618872    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z77jj\" (UniqueName: \"kubernetes.io/projected/956b4ddd-1483-402a-9b72-47871d24e448-kube-api-access-z77jj\") pod \"payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.618997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.618926    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm4kp\" (UniqueName: \"kubernetes.io/projected/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-kube-api-access-gm4kp\") pod \"payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.619544 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.619518    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b4ddd-1483-402a-9b72-47871d24e448-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.629920 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.629896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z77jj\" (UniqueName: \"kubernetes.io/projected/956b4ddd-1483-402a-9b72-47871d24e448-kube-api-access-z77jj\") pod \"payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.720024 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.719997    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm4kp\" (UniqueName: \"kubernetes.io/projected/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-kube-api-access-gm4kp\") pod \"payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.720210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.720061    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.720722 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.720702    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.728431 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.728414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm4kp\" (UniqueName: \"kubernetes.io/projected/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-kube-api-access-gm4kp\") pod \"payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:16.783343 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.783306    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:16.874760 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:16.874728    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:17.070922 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.070896    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"]
Sep 10 17:30:17.072403 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:30:17.072375    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956b4ddd_1483_402a_9b72_47871d24e448.slice/crio-680466e34878d21759ee6ab0f487fc495f0ad5f180a3367de2021c76c2ba49eb WatchSource:0}: Error finding container 680466e34878d21759ee6ab0f487fc495f0ad5f180a3367de2021c76c2ba49eb: Status 404 returned error can't find the container with id 680466e34878d21759ee6ab0f487fc495f0ad5f180a3367de2021c76c2ba49eb
Sep 10 17:30:17.154474 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.154445    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"]
Sep 10 17:30:17.154645 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:30:17.154543    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860871a9_97e2_4c7c_9f0a_f5a0d7dd5993.slice/crio-c29d802e4b91b6cfcfa34d94608e5ac580159dca3982537aa1b9f0c57bff250d WatchSource:0}: Error finding container c29d802e4b91b6cfcfa34d94608e5ac580159dca3982537aa1b9f0c57bff250d: Status 404 returned error can't find the container with id c29d802e4b91b6cfcfa34d94608e5ac580159dca3982537aa1b9f0c57bff250d
Sep 10 17:30:17.719783 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.719743    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww" event={"ID":"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993","Type":"ContainerStarted","Data":"496649757b3844a94cf41379028a50584214f1d05c988d135a41eb9b858323d8"}
Sep 10 17:30:17.720220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.719809    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww" event={"ID":"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993","Type":"ContainerStarted","Data":"c29d802e4b91b6cfcfa34d94608e5ac580159dca3982537aa1b9f0c57bff250d"}
Sep 10 17:30:17.720220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.719840    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:17.721522 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.721493    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc" event={"ID":"956b4ddd-1483-402a-9b72-47871d24e448","Type":"ContainerStarted","Data":"196566a8fb5e4b83799e9391eb1f2f343af9664559b70636489c5f05b4faebd7"}
Sep 10 17:30:17.721642 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.721542    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc" event={"ID":"956b4ddd-1483-402a-9b72-47871d24e448","Type":"ContainerStarted","Data":"680466e34878d21759ee6ab0f487fc495f0ad5f180a3367de2021c76c2ba49eb"}
Sep 10 17:30:17.721642 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.721589    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:17.749987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.749942    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww" podStartSLOduration=1.749927078 podStartE2EDuration="1.749927078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:17.748845742 +0000 UTC m=+1102.968969678" watchObservedRunningTime="2026-09-10 17:30:17.749927078 +0000 UTC m=+1102.970051011"
Sep 10 17:30:17.778971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:17.778918    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc" podStartSLOduration=1.7788981910000001 podStartE2EDuration="1.778898191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:17.778787254 +0000 UTC m=+1102.998911189" watchObservedRunningTime="2026-09-10 17:30:17.778898191 +0000 UTC m=+1102.999022128"
Sep 10 17:30:28.730623 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:28.730580    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:30:28.735931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:28.735901    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:30:38.565799 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.565764    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"]
Sep 10 17:30:38.569431 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.569413    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.574370 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.574347    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"kube-root-ca.crt\""
Sep 10 17:30:38.575176 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.575147    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"default-dockercfg-w6hf7\""
Sep 10 17:30:38.575176 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.575161    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"auth-test-model-be583c-kserve-self-signed-certs\""
Sep 10 17:30:38.575334 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.575189    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"openshift-service-ca.crt\""
Sep 10 17:30:38.590626 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.590595    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"]
Sep 10 17:30:38.617435 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.617405    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tmp-dir\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.617572 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.617451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjlzz\" (UniqueName: \"kubernetes.io/projected/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-kube-api-access-kjlzz\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.617572 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.617478    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-home\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.617572 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.617535    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-dshm\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.617572 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.617556    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-model-cache\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.617746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.617580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tls-certs\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.718686 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.718633    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tmp-dir\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.718883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.718711    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjlzz\" (UniqueName: \"kubernetes.io/projected/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-kube-api-access-kjlzz\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.718883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.718736    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-home\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.718883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.718812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-dshm\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.718883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.718847    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-model-cache\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.719115 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.718887    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tls-certs\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.719172 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.719127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tmp-dir\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.719172 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.719152    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-home\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.719250 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.719199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-model-cache\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.721027 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.721008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-dshm\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.721285 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.721268    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tls-certs\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.727952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.727931    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjlzz\" (UniqueName: \"kubernetes.io/projected/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-kube-api-access-kjlzz\") pod \"auth-test-model-be583c-kserve-85847f459f-w76qd\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") " pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:38.880013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:38.879919    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:39.156820 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:39.156789    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"]
Sep 10 17:30:39.158917 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:30:39.158890    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc65eb00_82d1_4efd_a3e6_fc831b1c8567.slice/crio-c3510addd11e4b3312873063b4f3e50a06fd86a3bfa4c6beac76ab48abecb50c WatchSource:0}: Error finding container c3510addd11e4b3312873063b4f3e50a06fd86a3bfa4c6beac76ab48abecb50c: Status 404 returned error can't find the container with id c3510addd11e4b3312873063b4f3e50a06fd86a3bfa4c6beac76ab48abecb50c
Sep 10 17:30:39.834733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:39.834698    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd" event={"ID":"dc65eb00-82d1-4efd-a3e6-fc831b1c8567","Type":"ContainerStarted","Data":"c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9"}
Sep 10 17:30:39.835146 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:39.834746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd" event={"ID":"dc65eb00-82d1-4efd-a3e6-fc831b1c8567","Type":"ContainerStarted","Data":"c3510addd11e4b3312873063b4f3e50a06fd86a3bfa4c6beac76ab48abecb50c"}
Sep 10 17:30:39.858880 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:39.858822    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd" podStartSLOduration=1.858803794 podStartE2EDuration="1.858803794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:39.854593108 +0000 UTC m=+1125.074717040" watchObservedRunningTime="2026-09-10 17:30:39.858803794 +0000 UTC m=+1125.078927728"
Sep 10 17:30:46.047810 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.047758    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"]
Sep 10 17:30:46.052275 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.052247    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.056027 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.055992    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-edbcb5\""
Sep 10 17:30:46.056160 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.056049    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-edbcb5-dockercfg-2p6kr\""
Sep 10 17:30:46.084045 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.084013    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"]
Sep 10 17:30:46.125132 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.125090    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"]
Sep 10 17:30:46.128782 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.128754    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.148329 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.148295    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"]
Sep 10 17:30:46.200449 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.200408    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8jz\" (UniqueName: \"kubernetes.io/projected/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-kube-api-access-gs8jz\") pod \"payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.200623 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.200544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.200623 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.200573    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jwmz\" (UniqueName: \"kubernetes.io/projected/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-kube-api-access-5jwmz\") pod \"payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.200725 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.200636    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.302105 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.302012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.302105 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.302059    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8jz\" (UniqueName: \"kubernetes.io/projected/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-kube-api-access-gs8jz\") pod \"payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.302317 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.302125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.302317 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.302141    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jwmz\" (UniqueName: \"kubernetes.io/projected/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-kube-api-access-5jwmz\") pod \"payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.302769 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.302746    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.302851 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.302832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.312825 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.312803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8jz\" (UniqueName: \"kubernetes.io/projected/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-kube-api-access-gs8jz\") pod \"payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.312928 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.312803    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jwmz\" (UniqueName: \"kubernetes.io/projected/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-kube-api-access-5jwmz\") pod \"payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.363309 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.363263    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.439299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.439028    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.643535 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:30:46.643499    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad48b28e_bcac_460c_a6b2_4774ed4d3b9d.slice/crio-27cb5386006a9cf0f7f776e3b60877d1834f3b5631c1bb13ea4539f6534d9d70 WatchSource:0}: Error finding container 27cb5386006a9cf0f7f776e3b60877d1834f3b5631c1bb13ea4539f6534d9d70: Status 404 returned error can't find the container with id 27cb5386006a9cf0f7f776e3b60877d1834f3b5631c1bb13ea4539f6534d9d70
Sep 10 17:30:46.644274 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.644246    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"]
Sep 10 17:30:46.729773 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.729727    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"]
Sep 10 17:30:46.732589 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:30:46.732564    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c41ae83_0b1c_4ed2_a346_34614f6fcfae.slice/crio-b18a72c168b1f9ca6b8fe762cb49b7b0c858980cdc399bc1e41ce5a3d876d3e8 WatchSource:0}: Error finding container b18a72c168b1f9ca6b8fe762cb49b7b0c858980cdc399bc1e41ce5a3d876d3e8: Status 404 returned error can't find the container with id b18a72c168b1f9ca6b8fe762cb49b7b0c858980cdc399bc1e41ce5a3d876d3e8
Sep 10 17:30:46.878986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.878931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7" event={"ID":"3c41ae83-0b1c-4ed2-a346-34614f6fcfae","Type":"ContainerStarted","Data":"0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7"}
Sep 10 17:30:46.879177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.879005    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7" event={"ID":"3c41ae83-0b1c-4ed2-a346-34614f6fcfae","Type":"ContainerStarted","Data":"b18a72c168b1f9ca6b8fe762cb49b7b0c858980cdc399bc1e41ce5a3d876d3e8"}
Sep 10 17:30:46.879177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.879036    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:30:46.881059 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.881026    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m" event={"ID":"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d","Type":"ContainerStarted","Data":"3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a"}
Sep 10 17:30:46.881185 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.881079    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m" event={"ID":"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d","Type":"ContainerStarted","Data":"27cb5386006a9cf0f7f776e3b60877d1834f3b5631c1bb13ea4539f6534d9d70"}
Sep 10 17:30:46.881229 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.881196    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:46.908515 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.908405    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7" podStartSLOduration=0.908390918 podStartE2EDuration="908.390918ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:46.906179422 +0000 UTC m=+1132.126303344" watchObservedRunningTime="2026-09-10 17:30:46.908390918 +0000 UTC m=+1132.128514907"
Sep 10 17:30:46.938468 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:46.938244    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m" podStartSLOduration=0.938223234 podStartE2EDuration="938.223234ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:30:46.936964082 +0000 UTC m=+1132.157088026" watchObservedRunningTime="2026-09-10 17:30:46.938223234 +0000 UTC m=+1132.158347184"
Sep 10 17:30:48.880619 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:48.880577    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:48.880619 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:48.880630    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:48.894491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:48.894451    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:48.906002 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:48.905972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:30:57.888279 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:57.888243    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:30:57.888726 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:30:57.888336    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:31:12.548464 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.548432    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"]
Sep 10 17:31:12.552266 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.552250    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.557301 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.557255    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"default-dockercfg-psk95\""
Sep 10 17:31:12.557422 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.557302    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"kube-root-ca.crt\""
Sep 10 17:31:12.557422 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.557357    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"openshift-service-ca.crt\""
Sep 10 17:31:12.557422 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.557383    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"auth-test-model-6806f1-kserve-self-signed-certs\""
Sep 10 17:31:12.566349 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.566323    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"]
Sep 10 17:31:12.666083 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.666039    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cvb\" (UniqueName: \"kubernetes.io/projected/76c466bc-66dc-4162-a780-c4f361ebd783-kube-api-access-v5cvb\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.666259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.666174    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-model-cache\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.666259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.666210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c466bc-66dc-4162-a780-c4f361ebd783-tls-certs\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.666259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.666240    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-dshm\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.666396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.666270    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-home\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.666396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.666362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-tmp-dir\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767125 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767091    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-tmp-dir\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767166    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cvb\" (UniqueName: \"kubernetes.io/projected/76c466bc-66dc-4162-a780-c4f361ebd783-kube-api-access-v5cvb\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767239    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-model-cache\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767267    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c466bc-66dc-4162-a780-c4f361ebd783-tls-certs\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-dshm\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767544 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767341    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-home\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767544 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767514    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-tmp-dir\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767608    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-model-cache\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.767737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.767705    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-home\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.769504 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.769480    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-dshm\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.769812 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.769793    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c466bc-66dc-4162-a780-c4f361ebd783-tls-certs\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.779933 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.779909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cvb\" (UniqueName: \"kubernetes.io/projected/76c466bc-66dc-4162-a780-c4f361ebd783-kube-api-access-v5cvb\") pod \"auth-test-model-6806f1-kserve-6f468d7f87-p7xzh\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:12.861841 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:12.861776    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:13.137650 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:13.137622    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"]
Sep 10 17:31:13.139964 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:31:13.139933    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76c466bc_66dc_4162_a780_c4f361ebd783.slice/crio-e37bcff82595e611bfa6c7021a62862c8bface3d91a4f078ea604dbcefea5780 WatchSource:0}: Error finding container e37bcff82595e611bfa6c7021a62862c8bface3d91a4f078ea604dbcefea5780: Status 404 returned error can't find the container with id e37bcff82595e611bfa6c7021a62862c8bface3d91a4f078ea604dbcefea5780
Sep 10 17:31:14.029619 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:14.029582    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh" event={"ID":"76c466bc-66dc-4162-a780-c4f361ebd783","Type":"ContainerStarted","Data":"4a6f5734d6bda4368c260fac0d203a963453eb3d95e63af02774215478491327"}
Sep 10 17:31:14.029619 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:14.029625    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh" event={"ID":"76c466bc-66dc-4162-a780-c4f361ebd783","Type":"ContainerStarted","Data":"e37bcff82595e611bfa6c7021a62862c8bface3d91a4f078ea604dbcefea5780"}
Sep 10 17:31:14.052647 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:14.052600    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh" podStartSLOduration=2.052584559 podStartE2EDuration="2.052584559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:14.0514376 +0000 UTC m=+1159.271561553" watchObservedRunningTime="2026-09-10 17:31:14.052584559 +0000 UTC m=+1159.272708516"
Sep 10 17:31:22.862119 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:22.862026    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:22.862119 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:22.862101    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:22.875004 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:22.874970    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:23.095064 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:23.095032    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:31:58.558977 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.558943    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"]
Sep 10 17:31:58.562735 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.562710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.566302 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.566253    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9d28f397\"/\"kube-root-ca.crt\""
Sep 10 17:31:58.567282 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.567254    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9d28f397\"/\"e2e-lifecycle-model-9d28f397-kserve-self-signed-certs\""
Sep 10 17:31:58.567282 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.567271    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9d28f397\"/\"openshift-service-ca.crt\""
Sep 10 17:31:58.567473 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.567275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9d28f397\"/\"default-dockercfg-nsbd8\""
Sep 10 17:31:58.580635 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.580612    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"]
Sep 10 17:31:58.713614 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.713579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ngl9\" (UniqueName: \"kubernetes.io/projected/31895234-1dde-4f4e-9338-3cbefa13f7de-kube-api-access-8ngl9\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.713817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.713634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-home\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.713817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.713688    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-dshm\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.713904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.713822    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-tmp-dir\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.713904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.713887    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31895234-1dde-4f4e-9338-3cbefa13f7de-tls-certs\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.713970 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.713920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-model-cache\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.814858 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.814777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31895234-1dde-4f4e-9338-3cbefa13f7de-tls-certs\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.814858 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.814819    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-model-cache\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.814858 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.814854    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ngl9\" (UniqueName: \"kubernetes.io/projected/31895234-1dde-4f4e-9338-3cbefa13f7de-kube-api-access-8ngl9\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.815166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.814887    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-home\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.815166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.814908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-dshm\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.815166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.815004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-tmp-dir\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.815335 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.815312    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-model-cache\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.815396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.815337    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-home\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.815396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.815387    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-tmp-dir\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.817237 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.817214    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-dshm\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.817464 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.817447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31895234-1dde-4f4e-9338-3cbefa13f7de-tls-certs\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.823761 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.823739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ngl9\" (UniqueName: \"kubernetes.io/projected/31895234-1dde-4f4e-9338-3cbefa13f7de-kube-api-access-8ngl9\") pod \"e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") " pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:58.873645 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:58.873607    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:31:59.151204 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:59.151176    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"]
Sep 10 17:31:59.153023 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:31:59.152991    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31895234_1dde_4f4e_9338_3cbefa13f7de.slice/crio-e273971e21ccf831ac069fa22176848528a3bf50fd5e3601ab51787e285912a7 WatchSource:0}: Error finding container e273971e21ccf831ac069fa22176848528a3bf50fd5e3601ab51787e285912a7: Status 404 returned error can't find the container with id e273971e21ccf831ac069fa22176848528a3bf50fd5e3601ab51787e285912a7
Sep 10 17:31:59.274187 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:59.274148    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz" event={"ID":"31895234-1dde-4f4e-9338-3cbefa13f7de","Type":"ContainerStarted","Data":"83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5"}
Sep 10 17:31:59.274187 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:59.274186    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz" event={"ID":"31895234-1dde-4f4e-9338-3cbefa13f7de","Type":"ContainerStarted","Data":"e273971e21ccf831ac069fa22176848528a3bf50fd5e3601ab51787e285912a7"}
Sep 10 17:31:59.292310 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:31:59.292263    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz" podStartSLOduration=1.2922473939999999 podStartE2EDuration="1.292247394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:31:59.290840119 +0000 UTC m=+1204.510964052" watchObservedRunningTime="2026-09-10 17:31:59.292247394 +0000 UTC m=+1204.512371331"
Sep 10 17:32:08.874709 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:08.874655    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:32:08.875190 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:08.874722    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:32:08.888406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:08.888377    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:32:09.337989 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:09.337959    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:32:36.259267 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.259230    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"]
Sep 10 17:32:36.263154 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.263134    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:36.265890 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.265857    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:32:36.266086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.265906    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:32:36.273162 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.273134    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"]
Sep 10 17:32:36.380188 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.380143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d8bce6-6247-42d7-b67c-e810825730a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f\" (UID: \"88d8bce6-6247-42d7-b67c-e810825730a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:36.481380 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.481350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d8bce6-6247-42d7-b67c-e810825730a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f\" (UID: \"88d8bce6-6247-42d7-b67c-e810825730a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:36.482205 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.482184    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d8bce6-6247-42d7-b67c-e810825730a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f\" (UID: \"88d8bce6-6247-42d7-b67c-e810825730a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:36.576539 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.576450    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:36.851266 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:36.851238    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"]
Sep 10 17:32:36.852953 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:32:36.852926    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d8bce6_6247_42d7_b67c_e810825730a1.slice/crio-a5ccee3b81bc37c135bc69db38896070a53751fb2eac8f62a45e6f282e3a5bbe WatchSource:0}: Error finding container a5ccee3b81bc37c135bc69db38896070a53751fb2eac8f62a45e6f282e3a5bbe: Status 404 returned error can't find the container with id a5ccee3b81bc37c135bc69db38896070a53751fb2eac8f62a45e6f282e3a5bbe
Sep 10 17:32:37.492931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:37.492893    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f" event={"ID":"88d8bce6-6247-42d7-b67c-e810825730a1","Type":"ContainerStarted","Data":"c1d521b7afc0001cd27a33177c0e810ce45243f089f473f7dd1150906132422c"}
Sep 10 17:32:37.493347 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:37.492956    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f" event={"ID":"88d8bce6-6247-42d7-b67c-e810825730a1","Type":"ContainerStarted","Data":"a5ccee3b81bc37c135bc69db38896070a53751fb2eac8f62a45e6f282e3a5bbe"}
Sep 10 17:32:37.515147 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:37.515093    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f" podStartSLOduration=1.515076264 podStartE2EDuration="1.515076264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:32:37.512835247 +0000 UTC m=+1242.732959179" watchObservedRunningTime="2026-09-10 17:32:37.515076264 +0000 UTC m=+1242.735200196"
Sep 10 17:32:38.499854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:38.499819    2556 generic.go:356] "Generic (PLEG): container finished" podID="88d8bce6-6247-42d7-b67c-e810825730a1" containerID="c1d521b7afc0001cd27a33177c0e810ce45243f089f473f7dd1150906132422c" exitCode=0
Sep 10 17:32:38.500239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:38.499904    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f" event={"ID":"88d8bce6-6247-42d7-b67c-e810825730a1","Type":"ContainerDied","Data":"c1d521b7afc0001cd27a33177c0e810ce45243f089f473f7dd1150906132422c"}
Sep 10 17:32:39.635130 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:39.635104    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:39.715621 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:39.715590    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d8bce6-6247-42d7-b67c-e810825730a1-maas-api-service-ca\") pod \"88d8bce6-6247-42d7-b67c-e810825730a1\" (UID: \"88d8bce6-6247-42d7-b67c-e810825730a1\") "
Sep 10 17:32:39.715969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:39.715946    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d8bce6-6247-42d7-b67c-e810825730a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88d8bce6-6247-42d7-b67c-e810825730a1" (UID: "88d8bce6-6247-42d7-b67c-e810825730a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:32:39.817255 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:39.817169    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88d8bce6-6247-42d7-b67c-e810825730a1-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:32:40.517148 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:40.517110    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f" event={"ID":"88d8bce6-6247-42d7-b67c-e810825730a1","Type":"ContainerDied","Data":"a5ccee3b81bc37c135bc69db38896070a53751fb2eac8f62a45e6f282e3a5bbe"}
Sep 10 17:32:40.517148 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:40.517140    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"
Sep 10 17:32:40.517376 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:40.517192    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ccee3b81bc37c135bc69db38896070a53751fb2eac8f62a45e6f282e3a5bbe"
Sep 10 17:32:52.446068 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:52.446031    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"]
Sep 10 17:32:52.448759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:52.447903    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9d28f397-s4b5f"]
Sep 10 17:32:53.271790 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:53.271754    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88d8bce6-6247-42d7-b67c-e810825730a1" path="/var/lib/kubelet/pods/88d8bce6-6247-42d7-b67c-e810825730a1/volumes"
Sep 10 17:32:55.347050 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:32:55.347019    2556 scope.go:117] "RemoveContainer" containerID="f45d59e636d3049ebc1fe0a9d02bfb7d66d5d7004932edccb387d6c1392b1c09"
Sep 10 17:33:00.690087 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:00.690053    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"]
Sep 10 17:33:15.709810 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:15.709772    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz" podUID="31895234-1dde-4f4e-9338-3cbefa13f7de" containerName="main" containerID="cri-o://83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5" gracePeriod=45
Sep 10 17:33:15.971804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:15.971776    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:33:16.071490 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071456    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-home\") pod \"31895234-1dde-4f4e-9338-3cbefa13f7de\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") "
Sep 10 17:33:16.071708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071570    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-dshm\") pod \"31895234-1dde-4f4e-9338-3cbefa13f7de\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") "
Sep 10 17:33:16.071708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071601    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ngl9\" (UniqueName: \"kubernetes.io/projected/31895234-1dde-4f4e-9338-3cbefa13f7de-kube-api-access-8ngl9\") pod \"31895234-1dde-4f4e-9338-3cbefa13f7de\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") "
Sep 10 17:33:16.071708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071636    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-tmp-dir\") pod \"31895234-1dde-4f4e-9338-3cbefa13f7de\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") "
Sep 10 17:33:16.071708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071655    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31895234-1dde-4f4e-9338-3cbefa13f7de-tls-certs\") pod \"31895234-1dde-4f4e-9338-3cbefa13f7de\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") "
Sep 10 17:33:16.071945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071720    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-model-cache\") pod \"31895234-1dde-4f4e-9338-3cbefa13f7de\" (UID: \"31895234-1dde-4f4e-9338-3cbefa13f7de\") "
Sep 10 17:33:16.071945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071806    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-home" (OuterVolumeSpecName: "home") pod "31895234-1dde-4f4e-9338-3cbefa13f7de" (UID: "31895234-1dde-4f4e-9338-3cbefa13f7de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:16.071945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.071913    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31895234-1dde-4f4e-9338-3cbefa13f7de" (UID: "31895234-1dde-4f4e-9338-3cbefa13f7de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:16.072095 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.072037    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-model-cache" (OuterVolumeSpecName: "model-cache") pod "31895234-1dde-4f4e-9338-3cbefa13f7de" (UID: "31895234-1dde-4f4e-9338-3cbefa13f7de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:16.072175 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.072156    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:16.072235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.072179    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:16.072235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.072193    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:16.073876 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.073848    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31895234-1dde-4f4e-9338-3cbefa13f7de-kube-api-access-8ngl9" (OuterVolumeSpecName: "kube-api-access-8ngl9") pod "31895234-1dde-4f4e-9338-3cbefa13f7de" (UID: "31895234-1dde-4f4e-9338-3cbefa13f7de"). InnerVolumeSpecName "kube-api-access-8ngl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:16.073876 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.073861    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31895234-1dde-4f4e-9338-3cbefa13f7de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31895234-1dde-4f4e-9338-3cbefa13f7de" (UID: "31895234-1dde-4f4e-9338-3cbefa13f7de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:16.074024 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.073876    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-dshm" (OuterVolumeSpecName: "dshm") pod "31895234-1dde-4f4e-9338-3cbefa13f7de" (UID: "31895234-1dde-4f4e-9338-3cbefa13f7de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:16.173358 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.173315    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31895234-1dde-4f4e-9338-3cbefa13f7de-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:16.173358 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.173344    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31895234-1dde-4f4e-9338-3cbefa13f7de-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:16.173358 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.173353    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ngl9\" (UniqueName: \"kubernetes.io/projected/31895234-1dde-4f4e-9338-3cbefa13f7de-kube-api-access-8ngl9\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:16.707947 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.707883    2556 generic.go:356] "Generic (PLEG): container finished" podID="31895234-1dde-4f4e-9338-3cbefa13f7de" containerID="83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5" exitCode=0
Sep 10 17:33:16.707947 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.707943    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz" event={"ID":"31895234-1dde-4f4e-9338-3cbefa13f7de","Type":"ContainerDied","Data":"83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5"}
Sep 10 17:33:16.708164 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.707979    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz" event={"ID":"31895234-1dde-4f4e-9338-3cbefa13f7de","Type":"ContainerDied","Data":"e273971e21ccf831ac069fa22176848528a3bf50fd5e3601ab51787e285912a7"}
Sep 10 17:33:16.708164 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.708000    2556 scope.go:117] "RemoveContainer" containerID="83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5"
Sep 10 17:33:16.708164 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.707926    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"
Sep 10 17:33:16.717452 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.717283    2556 scope.go:117] "RemoveContainer" containerID="83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5"
Sep 10 17:33:16.717675 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:33:16.717541    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5\": container with ID starting with 83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5 not found: ID does not exist" containerID="83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5"
Sep 10 17:33:16.717675 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.717563    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5"} err="failed to get container status \"83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5\": rpc error: code = NotFound desc = could not find container \"83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5\": container with ID starting with 83ade32b43032315d9573127c111ca655f777f0fc920a0680b5de4031b0049b5 not found: ID does not exist"
Sep 10 17:33:16.731674 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.731636    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"]
Sep 10 17:33:16.735758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:16.735735    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9d28f397/e2e-lifecycle-model-9d28f397-kserve-5f5f98979f-9smkz"]
Sep 10 17:33:17.272557 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:17.272522    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31895234-1dde-4f4e-9338-3cbefa13f7de" path="/var/lib/kubelet/pods/31895234-1dde-4f4e-9338-3cbefa13f7de/volumes"
Sep 10 17:33:36.537120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:36.537079    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"]
Sep 10 17:33:37.733966 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:37.733933    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"]
Sep 10 17:33:51.557084 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.557022    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd" podUID="dc65eb00-82d1-4efd-a3e6-fc831b1c8567" containerName="main" containerID="cri-o://c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9" gracePeriod=45
Sep 10 17:33:51.813755 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.813699    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:33:51.894865 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.894829    2556 generic.go:356] "Generic (PLEG): container finished" podID="dc65eb00-82d1-4efd-a3e6-fc831b1c8567" containerID="c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9" exitCode=0
Sep 10 17:33:51.895062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.894870    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"
Sep 10 17:33:51.895062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.894917    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd" event={"ID":"dc65eb00-82d1-4efd-a3e6-fc831b1c8567","Type":"ContainerDied","Data":"c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9"}
Sep 10 17:33:51.895062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.894966    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd" event={"ID":"dc65eb00-82d1-4efd-a3e6-fc831b1c8567","Type":"ContainerDied","Data":"c3510addd11e4b3312873063b4f3e50a06fd86a3bfa4c6beac76ab48abecb50c"}
Sep 10 17:33:51.895062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.894989    2556 scope.go:117] "RemoveContainer" containerID="c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9"
Sep 10 17:33:51.903767 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.903752    2556 scope.go:117] "RemoveContainer" containerID="c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9"
Sep 10 17:33:51.904029 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:33:51.904012    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9\": container with ID starting with c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9 not found: ID does not exist" containerID="c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9"
Sep 10 17:33:51.904077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.904037    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9"} err="failed to get container status \"c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9\": rpc error: code = NotFound desc = could not find container \"c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9\": container with ID starting with c765a07474f1900ecc3d1f0525caf233fc61f59eed11bc7d58e846759ce3f3d9 not found: ID does not exist"
Sep 10 17:33:51.922094 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922074    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tls-certs\") pod \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") "
Sep 10 17:33:51.922181 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922142    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-home\") pod \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") "
Sep 10 17:33:51.922275 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922186    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-model-cache\") pod \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") "
Sep 10 17:33:51.922320 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922294    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjlzz\" (UniqueName: \"kubernetes.io/projected/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-kube-api-access-kjlzz\") pod \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") "
Sep 10 17:33:51.922371 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922321    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tmp-dir\") pod \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") "
Sep 10 17:33:51.922427 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922348    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-dshm\") pod \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\" (UID: \"dc65eb00-82d1-4efd-a3e6-fc831b1c8567\") "
Sep 10 17:33:51.922427 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922407    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-home" (OuterVolumeSpecName: "home") pod "dc65eb00-82d1-4efd-a3e6-fc831b1c8567" (UID: "dc65eb00-82d1-4efd-a3e6-fc831b1c8567"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:51.922540 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922491    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc65eb00-82d1-4efd-a3e6-fc831b1c8567" (UID: "dc65eb00-82d1-4efd-a3e6-fc831b1c8567"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:51.922608 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922582    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc65eb00-82d1-4efd-a3e6-fc831b1c8567" (UID: "dc65eb00-82d1-4efd-a3e6-fc831b1c8567"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:51.922988 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922969    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:51.922988 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922989    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:51.922988 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.922997    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:51.924417 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.924390    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-dshm" (OuterVolumeSpecName: "dshm") pod "dc65eb00-82d1-4efd-a3e6-fc831b1c8567" (UID: "dc65eb00-82d1-4efd-a3e6-fc831b1c8567"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:51.924533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.924416    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc65eb00-82d1-4efd-a3e6-fc831b1c8567" (UID: "dc65eb00-82d1-4efd-a3e6-fc831b1c8567"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:51.924533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:51.924426    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-kube-api-access-kjlzz" (OuterVolumeSpecName: "kube-api-access-kjlzz") pod "dc65eb00-82d1-4efd-a3e6-fc831b1c8567" (UID: "dc65eb00-82d1-4efd-a3e6-fc831b1c8567"). InnerVolumeSpecName "kube-api-access-kjlzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:52.024333 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.024294    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjlzz\" (UniqueName: \"kubernetes.io/projected/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-kube-api-access-kjlzz\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:52.024333 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.024326    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:52.024333 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.024338    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc65eb00-82d1-4efd-a3e6-fc831b1c8567-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:52.224355 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.224323    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"]
Sep 10 17:33:52.231010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.230986    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/auth-test-model-be583c-kserve-85847f459f-w76qd"]
Sep 10 17:33:52.755848 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.755788    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh" podUID="76c466bc-66dc-4162-a780-c4f361ebd783" containerName="main" containerID="cri-o://4a6f5734d6bda4368c260fac0d203a963453eb3d95e63af02774215478491327" gracePeriod=45
Sep 10 17:33:52.903161 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.903129    2556 generic.go:356] "Generic (PLEG): container finished" podID="76c466bc-66dc-4162-a780-c4f361ebd783" containerID="4a6f5734d6bda4368c260fac0d203a963453eb3d95e63af02774215478491327" exitCode=0
Sep 10 17:33:52.903322 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:52.903170    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh" event={"ID":"76c466bc-66dc-4162-a780-c4f361ebd783","Type":"ContainerDied","Data":"4a6f5734d6bda4368c260fac0d203a963453eb3d95e63af02774215478491327"}
Sep 10 17:33:53.016126 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.016068    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:33:53.134437 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134404    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-tmp-dir\") pod \"76c466bc-66dc-4162-a780-c4f361ebd783\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") "
Sep 10 17:33:53.134437 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134442    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-home\") pod \"76c466bc-66dc-4162-a780-c4f361ebd783\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") "
Sep 10 17:33:53.134692 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134510    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-model-cache\") pod \"76c466bc-66dc-4162-a780-c4f361ebd783\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") "
Sep 10 17:33:53.134692 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134580    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c466bc-66dc-4162-a780-c4f361ebd783-tls-certs\") pod \"76c466bc-66dc-4162-a780-c4f361ebd783\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") "
Sep 10 17:33:53.134692 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134616    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-dshm\") pod \"76c466bc-66dc-4162-a780-c4f361ebd783\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") "
Sep 10 17:33:53.134692 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134684    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5cvb\" (UniqueName: \"kubernetes.io/projected/76c466bc-66dc-4162-a780-c4f361ebd783-kube-api-access-v5cvb\") pod \"76c466bc-66dc-4162-a780-c4f361ebd783\" (UID: \"76c466bc-66dc-4162-a780-c4f361ebd783\") "
Sep 10 17:33:53.134891 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134658    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76c466bc-66dc-4162-a780-c4f361ebd783" (UID: "76c466bc-66dc-4162-a780-c4f361ebd783"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:53.134891 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134746    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-home" (OuterVolumeSpecName: "home") pod "76c466bc-66dc-4162-a780-c4f361ebd783" (UID: "76c466bc-66dc-4162-a780-c4f361ebd783"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:53.134891 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.134832    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-model-cache" (OuterVolumeSpecName: "model-cache") pod "76c466bc-66dc-4162-a780-c4f361ebd783" (UID: "76c466bc-66dc-4162-a780-c4f361ebd783"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:53.135064 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.135048    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:53.135122 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.135065    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:53.135122 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.135075    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:53.136796 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.136767    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-dshm" (OuterVolumeSpecName: "dshm") pod "76c466bc-66dc-4162-a780-c4f361ebd783" (UID: "76c466bc-66dc-4162-a780-c4f361ebd783"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:33:53.136908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.136827    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76c466bc-66dc-4162-a780-c4f361ebd783-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76c466bc-66dc-4162-a780-c4f361ebd783" (UID: "76c466bc-66dc-4162-a780-c4f361ebd783"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:33:53.136908 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.136840    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76c466bc-66dc-4162-a780-c4f361ebd783-kube-api-access-v5cvb" (OuterVolumeSpecName: "kube-api-access-v5cvb") pod "76c466bc-66dc-4162-a780-c4f361ebd783" (UID: "76c466bc-66dc-4162-a780-c4f361ebd783"). InnerVolumeSpecName "kube-api-access-v5cvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:33:53.236502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.236470    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76c466bc-66dc-4162-a780-c4f361ebd783-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:53.236502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.236496    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5cvb\" (UniqueName: \"kubernetes.io/projected/76c466bc-66dc-4162-a780-c4f361ebd783-kube-api-access-v5cvb\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:53.236502 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.236506    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76c466bc-66dc-4162-a780-c4f361ebd783-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:33:53.272127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.272057    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc65eb00-82d1-4efd-a3e6-fc831b1c8567" path="/var/lib/kubelet/pods/dc65eb00-82d1-4efd-a3e6-fc831b1c8567/volumes"
Sep 10 17:33:53.909276 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.909236    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh" event={"ID":"76c466bc-66dc-4162-a780-c4f361ebd783","Type":"ContainerDied","Data":"e37bcff82595e611bfa6c7021a62862c8bface3d91a4f078ea604dbcefea5780"}
Sep 10 17:33:53.909276 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.909268    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"
Sep 10 17:33:53.909827 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.909305    2556 scope.go:117] "RemoveContainer" containerID="4a6f5734d6bda4368c260fac0d203a963453eb3d95e63af02774215478491327"
Sep 10 17:33:53.927224 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.927196    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"]
Sep 10 17:33:53.932759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:53.932738    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/auth-test-model-6806f1-kserve-6f468d7f87-p7xzh"]
Sep 10 17:33:55.272452 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:33:55.272414    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76c466bc-66dc-4162-a780-c4f361ebd783" path="/var/lib/kubelet/pods/76c466bc-66dc-4162-a780-c4f361ebd783/volumes"
Sep 10 17:34:02.795060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795022    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"]
Sep 10 17:34:02.795471 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795458    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88d8bce6-6247-42d7-b67c-e810825730a1" containerName="revoke-keys"
Sep 10 17:34:02.795526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795472    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="88d8bce6-6247-42d7-b67c-e810825730a1" containerName="revoke-keys"
Sep 10 17:34:02.795526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795486    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76c466bc-66dc-4162-a780-c4f361ebd783" containerName="main"
Sep 10 17:34:02.795526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795491    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="76c466bc-66dc-4162-a780-c4f361ebd783" containerName="main"
Sep 10 17:34:02.795526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795507    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31895234-1dde-4f4e-9338-3cbefa13f7de" containerName="main"
Sep 10 17:34:02.795526 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795514    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="31895234-1dde-4f4e-9338-3cbefa13f7de" containerName="main"
Sep 10 17:34:02.795699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795532    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc65eb00-82d1-4efd-a3e6-fc831b1c8567" containerName="main"
Sep 10 17:34:02.795699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795538    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc65eb00-82d1-4efd-a3e6-fc831b1c8567" containerName="main"
Sep 10 17:34:02.795699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795591    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc65eb00-82d1-4efd-a3e6-fc831b1c8567" containerName="main"
Sep 10 17:34:02.795699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795599    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="31895234-1dde-4f4e-9338-3cbefa13f7de" containerName="main"
Sep 10 17:34:02.795699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795605    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="88d8bce6-6247-42d7-b67c-e810825730a1" containerName="revoke-keys"
Sep 10 17:34:02.795699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.795612    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="76c466bc-66dc-4162-a780-c4f361ebd783" containerName="main"
Sep 10 17:34:02.799804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.799786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:02.806532 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.806505    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"default-dockercfg-w6hf7\""
Sep 10 17:34:02.806715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.806539    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"kube-root-ca.crt\""
Sep 10 17:34:02.806715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.806685    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"sub-test-model-be583c-kserve-self-signed-certs\""
Sep 10 17:34:02.807142 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.806505    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"openshift-service-ca.crt\""
Sep 10 17:34:02.828583 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.828547    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"]
Sep 10 17:34:02.929482 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.929449    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-model-cache\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:02.929753 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.929494    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cz4t\" (UniqueName: \"kubernetes.io/projected/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-kube-api-access-6cz4t\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:02.929753 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.929603    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-dshm\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:02.929753 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.929649    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tls-certs\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:02.929753 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.929704    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tmp-dir\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:02.929965 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:02.929784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-home\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031317 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031281    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tls-certs\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031332    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tmp-dir\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-home\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031418    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-model-cache\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031443    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cz4t\" (UniqueName: \"kubernetes.io/projected/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-kube-api-access-6cz4t\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031735 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-dshm\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031869 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031841    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-home\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031946 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031874    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tmp-dir\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.031946 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.031890    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-model-cache\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.033765 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.033745    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-dshm\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.033987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.033966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tls-certs\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.052321 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.052256    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cz4t\" (UniqueName: \"kubernetes.io/projected/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-kube-api-access-6cz4t\") pod \"sub-test-model-be583c-kserve-6796579d59-9m5pv\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") " pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.109904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.109872    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:03.429065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.429031    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"]
Sep 10 17:34:03.430004 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:34:03.429980    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ae28de2_a0c1_4e74_a9e8_bc06c11fbf36.slice/crio-d9775929a4c9683e190b65167287e41667e010ed6b3b0e00ca47473216ad9157 WatchSource:0}: Error finding container d9775929a4c9683e190b65167287e41667e010ed6b3b0e00ca47473216ad9157: Status 404 returned error can't find the container with id d9775929a4c9683e190b65167287e41667e010ed6b3b0e00ca47473216ad9157
Sep 10 17:34:03.963750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.963706    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv" event={"ID":"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36","Type":"ContainerStarted","Data":"2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826"}
Sep 10 17:34:03.964108 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:03.963760    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv" event={"ID":"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36","Type":"ContainerStarted","Data":"d9775929a4c9683e190b65167287e41667e010ed6b3b0e00ca47473216ad9157"}
Sep 10 17:34:13.110512 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:13.110477    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:13.110923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:13.110525    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:13.123382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:13.123356    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:13.141854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:13.141807    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv" podStartSLOduration=11.141793682 podStartE2EDuration="11.141793682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:04.010354431 +0000 UTC m=+1329.230478365" watchObservedRunningTime="2026-09-10 17:34:13.141793682 +0000 UTC m=+1338.361917615"
Sep 10 17:34:14.028255 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:14.028223    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:34:25.757653 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.757574    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"]
Sep 10 17:34:25.761122 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.761104    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.764425 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.764402    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"openshift-service-ca.crt\""
Sep 10 17:34:25.765127 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.765096    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"default-dockercfg-psk95\""
Sep 10 17:34:25.765306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.765155    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"kube-root-ca.crt\""
Sep 10 17:34:25.765474 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.765458    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"sub-test-model-6806f1-kserve-self-signed-certs\""
Sep 10 17:34:25.773576 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.773554    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"]
Sep 10 17:34:25.847836 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.847798    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tmp-dir\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.848019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.847847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-dshm\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.848019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.847943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-model-cache\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.848019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.847978    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkvq\" (UniqueName: \"kubernetes.io/projected/b9f394ba-3b88-4aa2-9b8f-e7775d954247-kube-api-access-dkkvq\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.848019 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.848008    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tls-certs\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.848156 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.848069    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-home\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.948921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.948880    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-home\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.948939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tmp-dir\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.948973    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-dshm\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.949021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-model-cache\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.949053    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkvq\" (UniqueName: \"kubernetes.io/projected/b9f394ba-3b88-4aa2-9b8f-e7775d954247-kube-api-access-dkkvq\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949135 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.949081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tls-certs\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.949358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-home\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.949386    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tmp-dir\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.949626 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.949606    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-model-cache\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.951247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.951226    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-dshm\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.951564 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.951548    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tls-certs\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:25.958469 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:25.958448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkvq\" (UniqueName: \"kubernetes.io/projected/b9f394ba-3b88-4aa2-9b8f-e7775d954247-kube-api-access-dkkvq\") pod \"sub-test-model-6806f1-kserve-5877c7ff65-gpfxd\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:26.070802 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:26.070708    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:26.345914 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:26.345882    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"]
Sep 10 17:34:26.347457 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:34:26.347428    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9f394ba_3b88_4aa2_9b8f_e7775d954247.slice/crio-7c9f89a726fbcd26650d2475d12c2043218476fcd713e45e2a6f949cd384e309 WatchSource:0}: Error finding container 7c9f89a726fbcd26650d2475d12c2043218476fcd713e45e2a6f949cd384e309: Status 404 returned error can't find the container with id 7c9f89a726fbcd26650d2475d12c2043218476fcd713e45e2a6f949cd384e309
Sep 10 17:34:27.087198 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:27.087160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd" event={"ID":"b9f394ba-3b88-4aa2-9b8f-e7775d954247","Type":"ContainerStarted","Data":"7e4ddc8dcd6162a9b7e205c5a681bb71e5b95c223a85f24bed5e2f1f99936ae5"}
Sep 10 17:34:27.087642 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:27.087211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd" event={"ID":"b9f394ba-3b88-4aa2-9b8f-e7775d954247","Type":"ContainerStarted","Data":"7c9f89a726fbcd26650d2475d12c2043218476fcd713e45e2a6f949cd384e309"}
Sep 10 17:34:27.124180 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:27.124127    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd" podStartSLOduration=2.124112234 podStartE2EDuration="2.124112234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:34:27.122250284 +0000 UTC m=+1352.342374217" watchObservedRunningTime="2026-09-10 17:34:27.124112234 +0000 UTC m=+1352.344236167"
Sep 10 17:34:36.071278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:36.071216    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:36.071708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:36.071306    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:36.084150 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:36.084124    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:34:36.147494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:34:36.147463    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:35:13.959421 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:13.959385    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"]
Sep 10 17:35:15.272791 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:15.272761    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"]
Sep 10 17:35:28.978852 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:28.978791    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv" podUID="7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" containerName="main" containerID="cri-o://2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826" gracePeriod=45
Sep 10 17:35:29.230180 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.230122    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:35:29.256964 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.256935    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-dshm\") pod \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") "
Sep 10 17:35:29.257117 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257052    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tls-certs\") pod \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") "
Sep 10 17:35:29.257187 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257121    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-home\") pod \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") "
Sep 10 17:35:29.257247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257237    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-model-cache\") pod \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") "
Sep 10 17:35:29.257314 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257286    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tmp-dir\") pod \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") "
Sep 10 17:35:29.257384 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257318    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cz4t\" (UniqueName: \"kubernetes.io/projected/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-kube-api-access-6cz4t\") pod \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\" (UID: \"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36\") "
Sep 10 17:35:29.257384 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257344    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-home" (OuterVolumeSpecName: "home") pod "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" (UID: "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:29.257521 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257499    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" (UID: "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:29.257599 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257576    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" (UID: "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:29.257758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257739    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:29.257849 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257761    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:29.257849 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.257774    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:29.259178 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.259151    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-dshm" (OuterVolumeSpecName: "dshm") pod "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" (UID: "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:29.259286 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.259222    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" (UID: "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:35:29.259396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.259381    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-kube-api-access-6cz4t" (OuterVolumeSpecName: "kube-api-access-6cz4t") pod "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" (UID: "7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36"). InnerVolumeSpecName "kube-api-access-6cz4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:35:29.358496 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.358471    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cz4t\" (UniqueName: \"kubernetes.io/projected/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-kube-api-access-6cz4t\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:29.358496 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.358495    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:29.358656 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.358505    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:29.425830 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.425798    2556 generic.go:356] "Generic (PLEG): container finished" podID="7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" containerID="2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826" exitCode=0
Sep 10 17:35:29.425968 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.425844    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"
Sep 10 17:35:29.425968 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.425888    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv" event={"ID":"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36","Type":"ContainerDied","Data":"2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826"}
Sep 10 17:35:29.425968 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.425935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv" event={"ID":"7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36","Type":"ContainerDied","Data":"d9775929a4c9683e190b65167287e41667e010ed6b3b0e00ca47473216ad9157"}
Sep 10 17:35:29.425968 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.425959    2556 scope.go:117] "RemoveContainer" containerID="2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826"
Sep 10 17:35:29.434894 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.434873    2556 scope.go:117] "RemoveContainer" containerID="2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826"
Sep 10 17:35:29.435178 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:35:29.435155    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826\": container with ID starting with 2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826 not found: ID does not exist" containerID="2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826"
Sep 10 17:35:29.435238 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.435182    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826"} err="failed to get container status \"2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826\": rpc error: code = NotFound desc = could not find container \"2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826\": container with ID starting with 2df6e6578ce7e44c920fcd1f81a10fb143257eb479a6ad820c887a7a33d71826 not found: ID does not exist"
Sep 10 17:35:29.442918 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.442892    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"]
Sep 10 17:35:29.446897 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:29.446875    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/sub-test-model-be583c-kserve-6796579d59-9m5pv"]
Sep 10 17:35:30.291932 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.291890    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd" podUID="b9f394ba-3b88-4aa2-9b8f-e7775d954247" containerName="main" containerID="cri-o://7e4ddc8dcd6162a9b7e205c5a681bb71e5b95c223a85f24bed5e2f1f99936ae5" gracePeriod=45
Sep 10 17:35:30.441283 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.441249    2556 generic.go:356] "Generic (PLEG): container finished" podID="b9f394ba-3b88-4aa2-9b8f-e7775d954247" containerID="7e4ddc8dcd6162a9b7e205c5a681bb71e5b95c223a85f24bed5e2f1f99936ae5" exitCode=0
Sep 10 17:35:30.441474 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.441324    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd" event={"ID":"b9f394ba-3b88-4aa2-9b8f-e7775d954247","Type":"ContainerDied","Data":"7e4ddc8dcd6162a9b7e205c5a681bb71e5b95c223a85f24bed5e2f1f99936ae5"}
Sep 10 17:35:30.558950 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.558880    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:35:30.673030 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.672997    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tmp-dir\") pod \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") "
Sep 10 17:35:30.673222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673045    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-dshm\") pod \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") "
Sep 10 17:35:30.673222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673100    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-model-cache\") pod \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") "
Sep 10 17:35:30.673222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673123    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-home\") pod \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") "
Sep 10 17:35:30.673222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673151    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tls-certs\") pod \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") "
Sep 10 17:35:30.673222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673191    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkkvq\" (UniqueName: \"kubernetes.io/projected/b9f394ba-3b88-4aa2-9b8f-e7775d954247-kube-api-access-dkkvq\") pod \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\" (UID: \"b9f394ba-3b88-4aa2-9b8f-e7775d954247\") "
Sep 10 17:35:30.673494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673360    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9f394ba-3b88-4aa2-9b8f-e7775d954247" (UID: "b9f394ba-3b88-4aa2-9b8f-e7775d954247"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:30.673494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673391    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9f394ba-3b88-4aa2-9b8f-e7775d954247" (UID: "b9f394ba-3b88-4aa2-9b8f-e7775d954247"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:30.673494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673448    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-home" (OuterVolumeSpecName: "home") pod "b9f394ba-3b88-4aa2-9b8f-e7775d954247" (UID: "b9f394ba-3b88-4aa2-9b8f-e7775d954247"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:30.673852 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673831    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:30.673987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673881    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:30.673987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.673897    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:30.675394 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.675372    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-dshm" (OuterVolumeSpecName: "dshm") pod "b9f394ba-3b88-4aa2-9b8f-e7775d954247" (UID: "b9f394ba-3b88-4aa2-9b8f-e7775d954247"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:35:30.675394 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.675377    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9f394ba-3b88-4aa2-9b8f-e7775d954247" (UID: "b9f394ba-3b88-4aa2-9b8f-e7775d954247"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:35:30.675578 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.675563    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f394ba-3b88-4aa2-9b8f-e7775d954247-kube-api-access-dkkvq" (OuterVolumeSpecName: "kube-api-access-dkkvq") pod "b9f394ba-3b88-4aa2-9b8f-e7775d954247" (UID: "b9f394ba-3b88-4aa2-9b8f-e7775d954247"). InnerVolumeSpecName "kube-api-access-dkkvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:35:30.774817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.774776    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9f394ba-3b88-4aa2-9b8f-e7775d954247-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:30.774817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.774810    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkkvq\" (UniqueName: \"kubernetes.io/projected/b9f394ba-3b88-4aa2-9b8f-e7775d954247-kube-api-access-dkkvq\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:30.774817 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:30.774819    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9f394ba-3b88-4aa2-9b8f-e7775d954247-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:35:31.272962 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:31.272926    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" path="/var/lib/kubelet/pods/7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36/volumes"
Sep 10 17:35:31.450156 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:31.450120    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"
Sep 10 17:35:31.450612 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:31.450172    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd" event={"ID":"b9f394ba-3b88-4aa2-9b8f-e7775d954247","Type":"ContainerDied","Data":"7c9f89a726fbcd26650d2475d12c2043218476fcd713e45e2a6f949cd384e309"}
Sep 10 17:35:31.450612 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:31.450227    2556 scope.go:117] "RemoveContainer" containerID="7e4ddc8dcd6162a9b7e205c5a681bb71e5b95c223a85f24bed5e2f1f99936ae5"
Sep 10 17:35:31.472002 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:31.471980    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"]
Sep 10 17:35:31.474795 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:31.474773    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/sub-test-model-6806f1-kserve-5877c7ff65-gpfxd"]
Sep 10 17:35:32.080589 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.080554    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"]
Sep 10 17:35:32.080999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.080986    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f394ba-3b88-4aa2-9b8f-e7775d954247" containerName="main"
Sep 10 17:35:32.081052 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.081001    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f394ba-3b88-4aa2-9b8f-e7775d954247" containerName="main"
Sep 10 17:35:32.081052 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.081027    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" containerName="main"
Sep 10 17:35:32.081052 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.081036    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" containerName="main"
Sep 10 17:35:32.081166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.081115    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f394ba-3b88-4aa2-9b8f-e7775d954247" containerName="main"
Sep 10 17:35:32.081166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.081128    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ae28de2-a0c1-4e74-a9e8-bc06c11fbf36" containerName="main"
Sep 10 17:35:32.085771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.085753    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.089534 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.089510    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"openshift-service-ca.crt\""
Sep 10 17:35:32.090217 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.090199    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"kube-root-ca.crt\""
Sep 10 17:35:32.090465 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.090436    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"default-dockercfg-w6hf7\""
Sep 10 17:35:32.090612 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.090463    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-be583c\"/\"rate-test-model-be583c-kserve-self-signed-certs\""
Sep 10 17:35:32.097575 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.097554    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"]
Sep 10 17:35:32.186697 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.186634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tls-certs\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.186856 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.186710    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-dshm\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.186856 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.186743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-home\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.186856 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.186757    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-model-cache\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.186856 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.186826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvq89\" (UniqueName: \"kubernetes.io/projected/7cd0c05c-0b7d-4161-9550-fd09e75f7225-kube-api-access-pvq89\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.186992 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.186861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tmp-dir\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287336    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvq89\" (UniqueName: \"kubernetes.io/projected/7cd0c05c-0b7d-4161-9550-fd09e75f7225-kube-api-access-pvq89\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tmp-dir\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tls-certs\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-dshm\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287538    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-home\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287737 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-model-cache\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tmp-dir\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.287923 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-home\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.288089 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.287937    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-model-cache\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.289834 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.289815    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-dshm\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.290010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.289994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tls-certs\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.301973 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.301942    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvq89\" (UniqueName: \"kubernetes.io/projected/7cd0c05c-0b7d-4161-9550-fd09e75f7225-kube-api-access-pvq89\") pod \"rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") " pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.395624 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.395536    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:32.670053 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:32.670013    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"]
Sep 10 17:35:32.671341 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:35:32.671308    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cd0c05c_0b7d_4161_9550_fd09e75f7225.slice/crio-68c71f8ef750f5fa3a8247a3906e5c24c2a1f7d7666aadf8d74abccf000a93f6 WatchSource:0}: Error finding container 68c71f8ef750f5fa3a8247a3906e5c24c2a1f7d7666aadf8d74abccf000a93f6: Status 404 returned error can't find the container with id 68c71f8ef750f5fa3a8247a3906e5c24c2a1f7d7666aadf8d74abccf000a93f6
Sep 10 17:35:33.272646 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:33.272615    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f394ba-3b88-4aa2-9b8f-e7775d954247" path="/var/lib/kubelet/pods/b9f394ba-3b88-4aa2-9b8f-e7775d954247/volumes"
Sep 10 17:35:33.467182 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:33.467146    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl" event={"ID":"7cd0c05c-0b7d-4161-9550-fd09e75f7225","Type":"ContainerStarted","Data":"fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a"}
Sep 10 17:35:33.467333 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:33.467195    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl" event={"ID":"7cd0c05c-0b7d-4161-9550-fd09e75f7225","Type":"ContainerStarted","Data":"68c71f8ef750f5fa3a8247a3906e5c24c2a1f7d7666aadf8d74abccf000a93f6"}
Sep 10 17:35:42.396716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:42.396681    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:42.397192 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:42.396739    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:42.409187 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:42.409158    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:42.430088 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:42.430041    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl" podStartSLOduration=10.430027818 podStartE2EDuration="10.430027818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:33.489515038 +0000 UTC m=+1418.709638970" watchObservedRunningTime="2026-09-10 17:35:42.430027818 +0000 UTC m=+1427.650151751"
Sep 10 17:35:42.529486 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:42.529462    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:35:46.250763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.250729    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"]
Sep 10 17:35:46.254296 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.254271    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.257721 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.257695    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-65fac0\""
Sep 10 17:35:46.276174 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.276134    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"]
Sep 10 17:35:46.278191 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.278170    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"]
Sep 10 17:35:46.281937 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.281917    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.290583 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.290546    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-65fac0-dockercfg-sdrm8\""
Sep 10 17:35:46.299015 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.298446    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"]
Sep 10 17:35:46.330026 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.329979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxnr\" (UniqueName: \"kubernetes.io/projected/d0996259-34b8-4d50-b50a-86b49853e3ae-kube-api-access-shxnr\") pod \"payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.330222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.330047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0996259-34b8-4d50-b50a-86b49853e3ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.431218 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.431168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxnr\" (UniqueName: \"kubernetes.io/projected/d0996259-34b8-4d50-b50a-86b49853e3ae-kube-api-access-shxnr\") pod \"payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.431457 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.431432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0996259-34b8-4d50-b50a-86b49853e3ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.431606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.431587    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7rg\" (UniqueName: \"kubernetes.io/projected/6fd890d4-8c85-44e5-bf63-2cf2630d475a-kube-api-access-7n7rg\") pod \"payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.431707 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.431641    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd890d4-8c85-44e5-bf63-2cf2630d475a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.432063 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.432042    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0996259-34b8-4d50-b50a-86b49853e3ae-plugins-config-volume\") pod \"payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.445378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.445352    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxnr\" (UniqueName: \"kubernetes.io/projected/d0996259-34b8-4d50-b50a-86b49853e3ae-kube-api-access-shxnr\") pod \"payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") " pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.533002 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.532903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7rg\" (UniqueName: \"kubernetes.io/projected/6fd890d4-8c85-44e5-bf63-2cf2630d475a-kube-api-access-7n7rg\") pod \"payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.533002 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.532950    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd890d4-8c85-44e5-bf63-2cf2630d475a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.533759 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.533735    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd890d4-8c85-44e5-bf63-2cf2630d475a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.546076 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.546049    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7rg\" (UniqueName: \"kubernetes.io/projected/6fd890d4-8c85-44e5-bf63-2cf2630d475a-kube-api-access-7n7rg\") pod \"payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.565627 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.565593    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:46.595833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.595795    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:46.853050 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.853023    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"]
Sep 10 17:35:46.853469 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:35:46.853443    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0996259_34b8_4d50_b50a_86b49853e3ae.slice/crio-b52655a818f93b6164509a19333e90244bbb25c191ceb4826073b7dbd764d079 WatchSource:0}: Error finding container b52655a818f93b6164509a19333e90244bbb25c191ceb4826073b7dbd764d079: Status 404 returned error can't find the container with id b52655a818f93b6164509a19333e90244bbb25c191ceb4826073b7dbd764d079
Sep 10 17:35:46.892566 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:46.892544    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"]
Sep 10 17:35:46.894212 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:35:46.894180    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd890d4_8c85_44e5_bf63_2cf2630d475a.slice/crio-c1930ccdf899b2f620809526c19ce10c0a0ad3a63fae89c2c82baea5b4025a8a WatchSource:0}: Error finding container c1930ccdf899b2f620809526c19ce10c0a0ad3a63fae89c2c82baea5b4025a8a: Status 404 returned error can't find the container with id c1930ccdf899b2f620809526c19ce10c0a0ad3a63fae89c2c82baea5b4025a8a
Sep 10 17:35:47.549437 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.549245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs" event={"ID":"6fd890d4-8c85-44e5-bf63-2cf2630d475a","Type":"ContainerStarted","Data":"f2a2c84f39bcfcb2518a7aa48271974e5c9c5f727ac4f00603e171eb05a3b99a"}
Sep 10 17:35:47.549437 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.549319    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs" event={"ID":"6fd890d4-8c85-44e5-bf63-2cf2630d475a","Type":"ContainerStarted","Data":"c1930ccdf899b2f620809526c19ce10c0a0ad3a63fae89c2c82baea5b4025a8a"}
Sep 10 17:35:47.549437 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.549352    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:47.551313 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.551285    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns" event={"ID":"d0996259-34b8-4d50-b50a-86b49853e3ae","Type":"ContainerStarted","Data":"7aebdaa40658d179d21fb28e8b6743183690e48ab6972d0a8611cdd9c3653c27"}
Sep 10 17:35:47.551438 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.551343    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:35:47.551438 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.551369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns" event={"ID":"d0996259-34b8-4d50-b50a-86b49853e3ae","Type":"ContainerStarted","Data":"b52655a818f93b6164509a19333e90244bbb25c191ceb4826073b7dbd764d079"}
Sep 10 17:35:47.573393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.573343    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs" podStartSLOduration=1.573327849 podStartE2EDuration="1.573327849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:47.570885943 +0000 UTC m=+1432.791009869" watchObservedRunningTime="2026-09-10 17:35:47.573327849 +0000 UTC m=+1432.793451856"
Sep 10 17:35:47.595706 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:47.595632    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns" podStartSLOduration=1.595617659 podStartE2EDuration="1.595617659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:35:47.59337269 +0000 UTC m=+1432.813496623" watchObservedRunningTime="2026-09-10 17:35:47.595617659 +0000 UTC m=+1432.815741632"
Sep 10 17:35:55.396035 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:55.396002    2556 scope.go:117] "RemoveContainer" containerID="958094f8c08228e54b99e66d2648fa0573e55d2e593688eed834af77d2c9aaca"
Sep 10 17:35:55.404498 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:55.404479    2556 scope.go:117] "RemoveContainer" containerID="655f225faf1753d959e331237616e157ef5e24f82db6a6cd47ca0584201ed12c"
Sep 10 17:35:58.558735 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:58.558700    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:35:58.559156 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:35:58.558772    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:36:11.849538 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:11.849500    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"]
Sep 10 17:36:11.853425 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:11.853399    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:11.856289 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:11.856260    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:36:11.856414 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:11.856337    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:36:11.860623 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:11.860594    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"]
Sep 10 17:36:11.990547 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:11.990506    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94b1313-5f28-4948-b58c-41ed95202090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl\" (UID: \"b94b1313-5f28-4948-b58c-41ed95202090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:12.091096 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:12.091056    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94b1313-5f28-4948-b58c-41ed95202090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl\" (UID: \"b94b1313-5f28-4948-b58c-41ed95202090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:12.091761 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:12.091743    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94b1313-5f28-4948-b58c-41ed95202090-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl\" (UID: \"b94b1313-5f28-4948-b58c-41ed95202090\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:12.167057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:12.166960    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:12.441077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:12.441049    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"]
Sep 10 17:36:12.442936 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:36:12.442907    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94b1313_5f28_4948_b58c_41ed95202090.slice/crio-b0fcc69bb10e1d1d15ea44cfe520a7e9ad90a986447e656058bf7c246598a36d WatchSource:0}: Error finding container b0fcc69bb10e1d1d15ea44cfe520a7e9ad90a986447e656058bf7c246598a36d: Status 404 returned error can't find the container with id b0fcc69bb10e1d1d15ea44cfe520a7e9ad90a986447e656058bf7c246598a36d
Sep 10 17:36:12.693422 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:12.693340    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl" event={"ID":"b94b1313-5f28-4948-b58c-41ed95202090","Type":"ContainerStarted","Data":"b0fcc69bb10e1d1d15ea44cfe520a7e9ad90a986447e656058bf7c246598a36d"}
Sep 10 17:36:13.699406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:13.699370    2556 generic.go:356] "Generic (PLEG): container finished" podID="b94b1313-5f28-4948-b58c-41ed95202090" containerID="c6e2b732d7aeff527fed57b7840978a83bbee9828aeb0fc2661e62e7fed95c75" exitCode=0
Sep 10 17:36:13.699884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:13.699456    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl" event={"ID":"b94b1313-5f28-4948-b58c-41ed95202090","Type":"ContainerDied","Data":"c6e2b732d7aeff527fed57b7840978a83bbee9828aeb0fc2661e62e7fed95c75"}
Sep 10 17:36:14.839443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:14.839419    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:14.920407 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:14.920375    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94b1313-5f28-4948-b58c-41ed95202090-maas-api-service-ca\") pod \"b94b1313-5f28-4948-b58c-41ed95202090\" (UID: \"b94b1313-5f28-4948-b58c-41ed95202090\") "
Sep 10 17:36:14.920738 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:14.920716    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94b1313-5f28-4948-b58c-41ed95202090-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b94b1313-5f28-4948-b58c-41ed95202090" (UID: "b94b1313-5f28-4948-b58c-41ed95202090"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:36:15.022024 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:15.021992    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b94b1313-5f28-4948-b58c-41ed95202090-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:36:15.711472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:15.711387    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl" event={"ID":"b94b1313-5f28-4948-b58c-41ed95202090","Type":"ContainerDied","Data":"b0fcc69bb10e1d1d15ea44cfe520a7e9ad90a986447e656058bf7c246598a36d"}
Sep 10 17:36:15.711472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:15.711408    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"
Sep 10 17:36:15.711472 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:15.711438    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0fcc69bb10e1d1d15ea44cfe520a7e9ad90a986447e656058bf7c246598a36d"
Sep 10 17:36:18.857901 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.857861    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"]
Sep 10 17:36:18.858400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.858328    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b94b1313-5f28-4948-b58c-41ed95202090" containerName="revoke-keys"
Sep 10 17:36:18.858400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.858339    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b94b1313-5f28-4948-b58c-41ed95202090" containerName="revoke-keys"
Sep 10 17:36:18.858477 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.858407    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b94b1313-5f28-4948-b58c-41ed95202090" containerName="revoke-keys"
Sep 10 17:36:18.861758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.861736    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:18.865670 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.865636    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"rate-test-model-6806f1-kserve-self-signed-certs\""
Sep 10 17:36:18.866101 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.866082    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"default-dockercfg-psk95\""
Sep 10 17:36:18.866484 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.866471    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"openshift-service-ca.crt\""
Sep 10 17:36:18.866961 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.866947    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-6806f1\"/\"kube-root-ca.crt\""
Sep 10 17:36:18.873394 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.873363    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"]
Sep 10 17:36:18.961953 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.961913    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-model-cache\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:18.962121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.961963    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-tmp-dir\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:18.962121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.962027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kt7n\" (UniqueName: \"kubernetes.io/projected/c690d92c-6194-45ac-a3cd-b6aea0150967-kube-api-access-7kt7n\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:18.962121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.962046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-home\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:18.962121 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.962065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-dshm\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:18.962268 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:18.962166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c690d92c-6194-45ac-a3cd-b6aea0150967-tls-certs\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063517 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063476    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c690d92c-6194-45ac-a3cd-b6aea0150967-tls-certs\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063543    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-model-cache\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063699    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-tmp-dir\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063791    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kt7n\" (UniqueName: \"kubernetes.io/projected/c690d92c-6194-45ac-a3cd-b6aea0150967-kube-api-access-7kt7n\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-home\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063835    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-dshm\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.063940 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.063904    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-model-cache\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.064022 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.064001    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-tmp-dir\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.064177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.064156    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-home\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.065930 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.065913    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-dshm\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.066055 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.066037    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c690d92c-6194-45ac-a3cd-b6aea0150967-tls-certs\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.073093 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.073067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kt7n\" (UniqueName: \"kubernetes.io/projected/c690d92c-6194-45ac-a3cd-b6aea0150967-kube-api-access-7kt7n\") pod \"rate-test-model-6806f1-kserve-54589f5787-9rmtw\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") " pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.171204 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.171119    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:19.445515 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.445485    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"]
Sep 10 17:36:19.447207 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:36:19.447175    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc690d92c_6194_45ac_a3cd_b6aea0150967.slice/crio-184ad1faa9dd2646ee63bf4f04934fe095e208b2e270f98516c63427ff6724d1 WatchSource:0}: Error finding container 184ad1faa9dd2646ee63bf4f04934fe095e208b2e270f98516c63427ff6724d1: Status 404 returned error can't find the container with id 184ad1faa9dd2646ee63bf4f04934fe095e208b2e270f98516c63427ff6724d1
Sep 10 17:36:19.734263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.734228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw" event={"ID":"c690d92c-6194-45ac-a3cd-b6aea0150967","Type":"ContainerStarted","Data":"daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408"}
Sep 10 17:36:19.734263 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.734275    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw" event={"ID":"c690d92c-6194-45ac-a3cd-b6aea0150967","Type":"ContainerStarted","Data":"184ad1faa9dd2646ee63bf4f04934fe095e208b2e270f98516c63427ff6724d1"}
Sep 10 17:36:19.753447 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:19.753399    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw" podStartSLOduration=1.753383637 podStartE2EDuration="1.753383637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:36:19.751999276 +0000 UTC m=+1464.972123210" watchObservedRunningTime="2026-09-10 17:36:19.753383637 +0000 UTC m=+1464.973507569"
Sep 10 17:36:27.686162 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:27.686121    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"]
Sep 10 17:36:27.688108 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:27.688085    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2549d0-b7pwl"]
Sep 10 17:36:29.171733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:29.171693    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:29.172134 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:29.171744    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:29.184298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:29.184267    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:29.272261 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:29.272232    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b94b1313-5f28-4948-b58c-41ed95202090" path="/var/lib/kubelet/pods/b94b1313-5f28-4948-b58c-41ed95202090/volumes"
Sep 10 17:36:29.811273 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:29.811235    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:36:50.633104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.633068    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"]
Sep 10 17:36:50.633760 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.633291    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns" podUID="d0996259-34b8-4d50-b50a-86b49853e3ae" containerName="payload-processing" containerID="cri-o://7aebdaa40658d179d21fb28e8b6743183690e48ab6972d0a8611cdd9c3653c27" gracePeriod=30
Sep 10 17:36:50.641880 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.640896    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"]
Sep 10 17:36:50.641880 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.641608    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs" podUID="6fd890d4-8c85-44e5-bf63-2cf2630d475a" containerName="payload-pre-processing" containerID="cri-o://f2a2c84f39bcfcb2518a7aa48271974e5c9c5f727ac4f00603e171eb05a3b99a" gracePeriod=30
Sep 10 17:36:50.914967 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.914937    2556 generic.go:356] "Generic (PLEG): container finished" podID="d0996259-34b8-4d50-b50a-86b49853e3ae" containerID="7aebdaa40658d179d21fb28e8b6743183690e48ab6972d0a8611cdd9c3653c27" exitCode=0
Sep 10 17:36:50.915120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.915017    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns" event={"ID":"d0996259-34b8-4d50-b50a-86b49853e3ae","Type":"ContainerDied","Data":"7aebdaa40658d179d21fb28e8b6743183690e48ab6972d0a8611cdd9c3653c27"}
Sep 10 17:36:50.915120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.915075    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns" event={"ID":"d0996259-34b8-4d50-b50a-86b49853e3ae","Type":"ContainerDied","Data":"b52655a818f93b6164509a19333e90244bbb25c191ceb4826073b7dbd764d079"}
Sep 10 17:36:50.915120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.915098    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b52655a818f93b6164509a19333e90244bbb25c191ceb4826073b7dbd764d079"
Sep 10 17:36:50.916595 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.916573    2556 generic.go:356] "Generic (PLEG): container finished" podID="6fd890d4-8c85-44e5-bf63-2cf2630d475a" containerID="f2a2c84f39bcfcb2518a7aa48271974e5c9c5f727ac4f00603e171eb05a3b99a" exitCode=0
Sep 10 17:36:50.916716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.916603    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs" event={"ID":"6fd890d4-8c85-44e5-bf63-2cf2630d475a","Type":"ContainerDied","Data":"f2a2c84f39bcfcb2518a7aa48271974e5c9c5f727ac4f00603e171eb05a3b99a"}
Sep 10 17:36:50.916716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.916685    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs" event={"ID":"6fd890d4-8c85-44e5-bf63-2cf2630d475a","Type":"ContainerDied","Data":"c1930ccdf899b2f620809526c19ce10c0a0ad3a63fae89c2c82baea5b4025a8a"}
Sep 10 17:36:50.916716 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.916711    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1930ccdf899b2f620809526c19ce10c0a0ad3a63fae89c2c82baea5b4025a8a"
Sep 10 17:36:50.926883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.926852    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:36:50.931839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:50.931818    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:36:51.079725 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.079686    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shxnr\" (UniqueName: \"kubernetes.io/projected/d0996259-34b8-4d50-b50a-86b49853e3ae-kube-api-access-shxnr\") pod \"d0996259-34b8-4d50-b50a-86b49853e3ae\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") "
Sep 10 17:36:51.079926 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.079752    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7n7rg\" (UniqueName: \"kubernetes.io/projected/6fd890d4-8c85-44e5-bf63-2cf2630d475a-kube-api-access-7n7rg\") pod \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") "
Sep 10 17:36:51.079926 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.079849    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd890d4-8c85-44e5-bf63-2cf2630d475a-plugins-config-volume\") pod \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\" (UID: \"6fd890d4-8c85-44e5-bf63-2cf2630d475a\") "
Sep 10 17:36:51.079926 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.079883    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0996259-34b8-4d50-b50a-86b49853e3ae-plugins-config-volume\") pod \"d0996259-34b8-4d50-b50a-86b49853e3ae\" (UID: \"d0996259-34b8-4d50-b50a-86b49853e3ae\") "
Sep 10 17:36:51.080313 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.080279    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0996259-34b8-4d50-b50a-86b49853e3ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0996259-34b8-4d50-b50a-86b49853e3ae" (UID: "d0996259-34b8-4d50-b50a-86b49853e3ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:36:51.080313 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.080279    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd890d4-8c85-44e5-bf63-2cf2630d475a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fd890d4-8c85-44e5-bf63-2cf2630d475a" (UID: "6fd890d4-8c85-44e5-bf63-2cf2630d475a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:36:51.081920 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.081898    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0996259-34b8-4d50-b50a-86b49853e3ae-kube-api-access-shxnr" (OuterVolumeSpecName: "kube-api-access-shxnr") pod "d0996259-34b8-4d50-b50a-86b49853e3ae" (UID: "d0996259-34b8-4d50-b50a-86b49853e3ae"). InnerVolumeSpecName "kube-api-access-shxnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:36:51.082000 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.081965    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd890d4-8c85-44e5-bf63-2cf2630d475a-kube-api-access-7n7rg" (OuterVolumeSpecName: "kube-api-access-7n7rg") pod "6fd890d4-8c85-44e5-bf63-2cf2630d475a" (UID: "6fd890d4-8c85-44e5-bf63-2cf2630d475a"). InnerVolumeSpecName "kube-api-access-7n7rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:36:51.181374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.181286    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd890d4-8c85-44e5-bf63-2cf2630d475a-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:36:51.181374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.181322    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0996259-34b8-4d50-b50a-86b49853e3ae-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:36:51.181374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.181334    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxnr\" (UniqueName: \"kubernetes.io/projected/d0996259-34b8-4d50-b50a-86b49853e3ae-kube-api-access-shxnr\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:36:51.181374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.181342    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7rg\" (UniqueName: \"kubernetes.io/projected/6fd890d4-8c85-44e5-bf63-2cf2630d475a-kube-api-access-7n7rg\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:36:51.921644 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.921610    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"
Sep 10 17:36:51.922058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.921615    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"
Sep 10 17:36:51.942208 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.942178    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"]
Sep 10 17:36:51.945758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.945733    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-65fac0-84774f9d6b-7j2xs"]
Sep 10 17:36:51.956792 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.956753    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"]
Sep 10 17:36:51.959443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:51.959420    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-65fac0-77d6ddbcf5-t2cns"]
Sep 10 17:36:53.272585 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:53.272548    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd890d4-8c85-44e5-bf63-2cf2630d475a" path="/var/lib/kubelet/pods/6fd890d4-8c85-44e5-bf63-2cf2630d475a/volumes"
Sep 10 17:36:53.272992 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:36:53.272962    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0996259-34b8-4d50-b50a-86b49853e3ae" path="/var/lib/kubelet/pods/d0996259-34b8-4d50-b50a-86b49853e3ae/volumes"
Sep 10 17:37:02.331566 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:02.331533    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"]
Sep 10 17:37:03.440501 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:03.440463    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"]
Sep 10 17:37:08.882298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:08.882252    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"]
Sep 10 17:37:08.882774 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:08.882570    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww" podUID="860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" containerName="payload-processing" containerID="cri-o://496649757b3844a94cf41379028a50584214f1d05c988d135a41eb9b858323d8" gracePeriod=30
Sep 10 17:37:08.893812 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:08.893391    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"]
Sep 10 17:37:08.893812 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:08.893641    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc" podUID="956b4ddd-1483-402a-9b72-47871d24e448" containerName="payload-pre-processing" containerID="cri-o://196566a8fb5e4b83799e9391eb1f2f343af9664559b70636489c5f05b4faebd7" gracePeriod=30
Sep 10 17:37:09.021904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.021869    2556 generic.go:356] "Generic (PLEG): container finished" podID="956b4ddd-1483-402a-9b72-47871d24e448" containerID="196566a8fb5e4b83799e9391eb1f2f343af9664559b70636489c5f05b4faebd7" exitCode=0
Sep 10 17:37:09.022016 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.021923    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc" event={"ID":"956b4ddd-1483-402a-9b72-47871d24e448","Type":"ContainerDied","Data":"196566a8fb5e4b83799e9391eb1f2f343af9664559b70636489c5f05b4faebd7"}
Sep 10 17:37:09.023804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.023779    2556 generic.go:356] "Generic (PLEG): container finished" podID="860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" containerID="496649757b3844a94cf41379028a50584214f1d05c988d135a41eb9b858323d8" exitCode=0
Sep 10 17:37:09.023907 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.023847    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww" event={"ID":"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993","Type":"ContainerDied","Data":"496649757b3844a94cf41379028a50584214f1d05c988d135a41eb9b858323d8"}
Sep 10 17:37:09.189904 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.189867    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:37:09.192428 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.192405    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:37:09.261475 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.261441    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-plugins-config-volume\") pod \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") "
Sep 10 17:37:09.261697 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.261498    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z77jj\" (UniqueName: \"kubernetes.io/projected/956b4ddd-1483-402a-9b72-47871d24e448-kube-api-access-z77jj\") pod \"956b4ddd-1483-402a-9b72-47871d24e448\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") "
Sep 10 17:37:09.261697 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.261529    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b4ddd-1483-402a-9b72-47871d24e448-plugins-config-volume\") pod \"956b4ddd-1483-402a-9b72-47871d24e448\" (UID: \"956b4ddd-1483-402a-9b72-47871d24e448\") "
Sep 10 17:37:09.261697 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.261603    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm4kp\" (UniqueName: \"kubernetes.io/projected/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-kube-api-access-gm4kp\") pod \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\" (UID: \"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993\") "
Sep 10 17:37:09.261970 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.261938    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" (UID: "860871a9-97e2-4c7c-9f0a-f5a0d7dd5993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:09.262044 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.262019    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956b4ddd-1483-402a-9b72-47871d24e448-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "956b4ddd-1483-402a-9b72-47871d24e448" (UID: "956b4ddd-1483-402a-9b72-47871d24e448"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:09.263721 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.263696    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956b4ddd-1483-402a-9b72-47871d24e448-kube-api-access-z77jj" (OuterVolumeSpecName: "kube-api-access-z77jj") pod "956b4ddd-1483-402a-9b72-47871d24e448" (UID: "956b4ddd-1483-402a-9b72-47871d24e448"). InnerVolumeSpecName "kube-api-access-z77jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:09.263845 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.263824    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-kube-api-access-gm4kp" (OuterVolumeSpecName: "kube-api-access-gm4kp") pod "860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" (UID: "860871a9-97e2-4c7c-9f0a-f5a0d7dd5993"). InnerVolumeSpecName "kube-api-access-gm4kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:09.363715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.363652    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm4kp\" (UniqueName: \"kubernetes.io/projected/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-kube-api-access-gm4kp\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:09.363715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.363706    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:09.363715 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.363723    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z77jj\" (UniqueName: \"kubernetes.io/projected/956b4ddd-1483-402a-9b72-47871d24e448-kube-api-access-z77jj\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:09.363943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.363738    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/956b4ddd-1483-402a-9b72-47871d24e448-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:09.993582 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.993543    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"]
Sep 10 17:37:09.994254 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994232    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" containerName="payload-processing"
Sep 10 17:37:09.994298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994259    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" containerName="payload-processing"
Sep 10 17:37:09.994298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994280    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956b4ddd-1483-402a-9b72-47871d24e448" containerName="payload-pre-processing"
Sep 10 17:37:09.994298 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994290    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="956b4ddd-1483-402a-9b72-47871d24e448" containerName="payload-pre-processing"
Sep 10 17:37:09.994397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994302    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd890d4-8c85-44e5-bf63-2cf2630d475a" containerName="payload-pre-processing"
Sep 10 17:37:09.994397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994312    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd890d4-8c85-44e5-bf63-2cf2630d475a" containerName="payload-pre-processing"
Sep 10 17:37:09.994397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994351    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0996259-34b8-4d50-b50a-86b49853e3ae" containerName="payload-processing"
Sep 10 17:37:09.994397 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994360    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0996259-34b8-4d50-b50a-86b49853e3ae" containerName="payload-processing"
Sep 10 17:37:09.994523 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994463    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" containerName="payload-processing"
Sep 10 17:37:09.994523 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994476    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd890d4-8c85-44e5-bf63-2cf2630d475a" containerName="payload-pre-processing"
Sep 10 17:37:09.994523 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994493    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="956b4ddd-1483-402a-9b72-47871d24e448" containerName="payload-pre-processing"
Sep 10 17:37:09.994523 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.994502    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0996259-34b8-4d50-b50a-86b49853e3ae" containerName="payload-processing"
Sep 10 17:37:09.998288 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:09.998265    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.002306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.002282    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-437dd7\""
Sep 10 17:37:10.004073 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.004054    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-437dd7-dockercfg-8mlld\""
Sep 10 17:37:10.013586 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.013563    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"]
Sep 10 17:37:10.033225 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.033201    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"
Sep 10 17:37:10.034305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.033321    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww" event={"ID":"860871a9-97e2-4c7c-9f0a-f5a0d7dd5993","Type":"ContainerDied","Data":"c29d802e4b91b6cfcfa34d94608e5ac580159dca3982537aa1b9f0c57bff250d"}
Sep 10 17:37:10.034305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.033962    2556 scope.go:117] "RemoveContainer" containerID="496649757b3844a94cf41379028a50584214f1d05c988d135a41eb9b858323d8"
Sep 10 17:37:10.036546 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.036464    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc" event={"ID":"956b4ddd-1483-402a-9b72-47871d24e448","Type":"ContainerDied","Data":"680466e34878d21759ee6ab0f487fc495f0ad5f180a3367de2021c76c2ba49eb"}
Sep 10 17:37:10.036720 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.036557    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"
Sep 10 17:37:10.052243 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.052219    2556 scope.go:117] "RemoveContainer" containerID="196566a8fb5e4b83799e9391eb1f2f343af9664559b70636489c5f05b4faebd7"
Sep 10 17:37:10.068339 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.068314    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"]
Sep 10 17:37:10.070893 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.070871    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3f6bb-2d1e-406e-990f-5a674180372f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.070958 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.070918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28n4\" (UniqueName: \"kubernetes.io/projected/5ed3f6bb-2d1e-406e-990f-5a674180372f-kube-api-access-n28n4\") pod \"payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.075227 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.075199    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-33b7bb-749bf8fcf4-qj5jc"]
Sep 10 17:37:10.099411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.099380    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"]
Sep 10 17:37:10.106775 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.106752    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-33b7bb-9f5ff9967-cx5ww"]
Sep 10 17:37:10.172243 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.172205    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3f6bb-2d1e-406e-990f-5a674180372f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.172411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.172264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28n4\" (UniqueName: \"kubernetes.io/projected/5ed3f6bb-2d1e-406e-990f-5a674180372f-kube-api-access-n28n4\") pod \"payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.173070 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.173050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3f6bb-2d1e-406e-990f-5a674180372f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.189056 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.189032    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28n4\" (UniqueName: \"kubernetes.io/projected/5ed3f6bb-2d1e-406e-990f-5a674180372f-kube-api-access-n28n4\") pod \"payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.309945 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.309857    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:10.593400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:10.593370    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"]
Sep 10 17:37:10.595906 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:37:10.595874    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed3f6bb_2d1e_406e_990f_5a674180372f.slice/crio-1ffae58af8c909283b293d9737139b1c5e3e85886794cad8ccd49ebb2ceb06bf WatchSource:0}: Error finding container 1ffae58af8c909283b293d9737139b1c5e3e85886794cad8ccd49ebb2ceb06bf: Status 404 returned error can't find the container with id 1ffae58af8c909283b293d9737139b1c5e3e85886794cad8ccd49ebb2ceb06bf
Sep 10 17:37:11.046364 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:11.046322    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs" event={"ID":"5ed3f6bb-2d1e-406e-990f-5a674180372f","Type":"ContainerStarted","Data":"e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4"}
Sep 10 17:37:11.046874 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:11.046395    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs" event={"ID":"5ed3f6bb-2d1e-406e-990f-5a674180372f","Type":"ContainerStarted","Data":"1ffae58af8c909283b293d9737139b1c5e3e85886794cad8ccd49ebb2ceb06bf"}
Sep 10 17:37:11.046874 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:11.046520    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:11.067018 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:11.066971    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs" podStartSLOduration=2.066955797 podStartE2EDuration="2.066955797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:11.065028389 +0000 UTC m=+1516.285152358" watchObservedRunningTime="2026-09-10 17:37:11.066955797 +0000 UTC m=+1516.287079729"
Sep 10 17:37:11.279166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:11.279128    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="860871a9-97e2-4c7c-9f0a-f5a0d7dd5993" path="/var/lib/kubelet/pods/860871a9-97e2-4c7c-9f0a-f5a0d7dd5993/volumes"
Sep 10 17:37:11.279510 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:11.279498    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956b4ddd-1483-402a-9b72-47871d24e448" path="/var/lib/kubelet/pods/956b4ddd-1483-402a-9b72-47871d24e448/volumes"
Sep 10 17:37:15.836947 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:15.836909    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"]
Sep 10 17:37:15.841860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:15.841839    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:15.844311 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:15.844287    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:37:15.844865 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:15.844840    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:37:15.850223 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:15.850191    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"]
Sep 10 17:37:15.925785 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:15.925737    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c36f87-d623-49d8-89de-10069981ba48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd\" (UID: \"29c36f87-d623-49d8-89de-10069981ba48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:16.027186 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:16.027145    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c36f87-d623-49d8-89de-10069981ba48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd\" (UID: \"29c36f87-d623-49d8-89de-10069981ba48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:16.027860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:16.027834    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c36f87-d623-49d8-89de-10069981ba48-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd\" (UID: \"29c36f87-d623-49d8-89de-10069981ba48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:16.155042 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:16.154942    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:16.429444 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:16.429416    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"]
Sep 10 17:37:16.431301 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:37:16.431268    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c36f87_d623_49d8_89de_10069981ba48.slice/crio-5b72486f5c947739e24d43cf2dc98b415b9ee6f63ff5734e1c4bfeaa89a54187 WatchSource:0}: Error finding container 5b72486f5c947739e24d43cf2dc98b415b9ee6f63ff5734e1c4bfeaa89a54187: Status 404 returned error can't find the container with id 5b72486f5c947739e24d43cf2dc98b415b9ee6f63ff5734e1c4bfeaa89a54187
Sep 10 17:37:17.089443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.089274    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd" event={"ID":"29c36f87-d623-49d8-89de-10069981ba48","Type":"ContainerStarted","Data":"003f0c9998b98e45b9912868421c4f5ff1402a56bee785f06559698f54144f7c"}
Sep 10 17:37:17.089919 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.089453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd" event={"ID":"29c36f87-d623-49d8-89de-10069981ba48","Type":"ContainerStarted","Data":"5b72486f5c947739e24d43cf2dc98b415b9ee6f63ff5734e1c4bfeaa89a54187"}
Sep 10 17:37:17.111720 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.111654    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd" podStartSLOduration=2.111640187 podStartE2EDuration="2.111640187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:17.108268867 +0000 UTC m=+1522.328392800" watchObservedRunningTime="2026-09-10 17:37:17.111640187 +0000 UTC m=+1522.331764119"
Sep 10 17:37:17.208523 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:37:17.208488    2556 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c36f87_d623_49d8_89de_10069981ba48.slice/crio-conmon-003f0c9998b98e45b9912868421c4f5ff1402a56bee785f06559698f54144f7c.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 17:37:17.350940 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.350822    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl" podUID="7cd0c05c-0b7d-4161-9550-fd09e75f7225" containerName="main" containerID="cri-o://fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a" gracePeriod=45
Sep 10 17:37:17.617280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.617214    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:37:17.743763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.743716    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-home\") pod \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") "
Sep 10 17:37:17.743939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.743796    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvq89\" (UniqueName: \"kubernetes.io/projected/7cd0c05c-0b7d-4161-9550-fd09e75f7225-kube-api-access-pvq89\") pod \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") "
Sep 10 17:37:17.743939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.743915    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-model-cache\") pod \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") "
Sep 10 17:37:17.744075 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.743949    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-dshm\") pod \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") "
Sep 10 17:37:17.744075 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.743969    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tls-certs\") pod \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") "
Sep 10 17:37:17.744075 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.743992    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tmp-dir\") pod \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\" (UID: \"7cd0c05c-0b7d-4161-9550-fd09e75f7225\") "
Sep 10 17:37:17.744075 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.744044    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-home" (OuterVolumeSpecName: "home") pod "7cd0c05c-0b7d-4161-9550-fd09e75f7225" (UID: "7cd0c05c-0b7d-4161-9550-fd09e75f7225"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:17.744305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.744277    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-model-cache" (OuterVolumeSpecName: "model-cache") pod "7cd0c05c-0b7d-4161-9550-fd09e75f7225" (UID: "7cd0c05c-0b7d-4161-9550-fd09e75f7225"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:17.744415 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.744381    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7cd0c05c-0b7d-4161-9550-fd09e75f7225" (UID: "7cd0c05c-0b7d-4161-9550-fd09e75f7225"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:17.744542 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.744432    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:17.744542 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.744449    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:17.746340 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.746315    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-dshm" (OuterVolumeSpecName: "dshm") pod "7cd0c05c-0b7d-4161-9550-fd09e75f7225" (UID: "7cd0c05c-0b7d-4161-9550-fd09e75f7225"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:17.746718 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.746689    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd0c05c-0b7d-4161-9550-fd09e75f7225-kube-api-access-pvq89" (OuterVolumeSpecName: "kube-api-access-pvq89") pod "7cd0c05c-0b7d-4161-9550-fd09e75f7225" (UID: "7cd0c05c-0b7d-4161-9550-fd09e75f7225"). InnerVolumeSpecName "kube-api-access-pvq89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:17.746718 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.746701    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7cd0c05c-0b7d-4161-9550-fd09e75f7225" (UID: "7cd0c05c-0b7d-4161-9550-fd09e75f7225"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:37:17.845724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.845687    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:17.845724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.845717    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:17.845724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.845727    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cd0c05c-0b7d-4161-9550-fd09e75f7225-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:17.845724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:17.845736    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvq89\" (UniqueName: \"kubernetes.io/projected/7cd0c05c-0b7d-4161-9550-fd09e75f7225-kube-api-access-pvq89\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:18.096129 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.096093    2556 generic.go:356] "Generic (PLEG): container finished" podID="29c36f87-d623-49d8-89de-10069981ba48" containerID="003f0c9998b98e45b9912868421c4f5ff1402a56bee785f06559698f54144f7c" exitCode=0
Sep 10 17:37:18.096603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.096182    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd" event={"ID":"29c36f87-d623-49d8-89de-10069981ba48","Type":"ContainerDied","Data":"003f0c9998b98e45b9912868421c4f5ff1402a56bee785f06559698f54144f7c"}
Sep 10 17:37:18.097520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.097499    2556 generic.go:356] "Generic (PLEG): container finished" podID="7cd0c05c-0b7d-4161-9550-fd09e75f7225" containerID="fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a" exitCode=0
Sep 10 17:37:18.097596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.097540    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"
Sep 10 17:37:18.097596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.097584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl" event={"ID":"7cd0c05c-0b7d-4161-9550-fd09e75f7225","Type":"ContainerDied","Data":"fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a"}
Sep 10 17:37:18.097695 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.097628    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl" event={"ID":"7cd0c05c-0b7d-4161-9550-fd09e75f7225","Type":"ContainerDied","Data":"68c71f8ef750f5fa3a8247a3906e5c24c2a1f7d7666aadf8d74abccf000a93f6"}
Sep 10 17:37:18.097695 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.097650    2556 scope.go:117] "RemoveContainer" containerID="fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a"
Sep 10 17:37:18.107256 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.107238    2556 scope.go:117] "RemoveContainer" containerID="fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a"
Sep 10 17:37:18.107518 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:37:18.107501    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a\": container with ID starting with fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a not found: ID does not exist" containerID="fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a"
Sep 10 17:37:18.107580 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.107526    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a"} err="failed to get container status \"fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a\": rpc error: code = NotFound desc = could not find container \"fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a\": container with ID starting with fb3f73e243c9e28e8cc35a5a0a383816db68c2f76d98dc1e2cf52e681968453a not found: ID does not exist"
Sep 10 17:37:18.123289 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.123266    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"]
Sep 10 17:37:18.125277 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.125253    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-be583c/rate-test-model-be583c-kserve-7cdfc8f4cf-8c5sl"]
Sep 10 17:37:18.460606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.460520    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw" podUID="c690d92c-6194-45ac-a3cd-b6aea0150967" containerName="main" containerID="cri-o://daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408" gracePeriod=45
Sep 10 17:37:18.737094 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.737058    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:37:18.855101 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855066    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-model-cache\") pod \"c690d92c-6194-45ac-a3cd-b6aea0150967\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") "
Sep 10 17:37:18.855305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855112    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-home\") pod \"c690d92c-6194-45ac-a3cd-b6aea0150967\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") "
Sep 10 17:37:18.855305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855162    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-tmp-dir\") pod \"c690d92c-6194-45ac-a3cd-b6aea0150967\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") "
Sep 10 17:37:18.855305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855233    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kt7n\" (UniqueName: \"kubernetes.io/projected/c690d92c-6194-45ac-a3cd-b6aea0150967-kube-api-access-7kt7n\") pod \"c690d92c-6194-45ac-a3cd-b6aea0150967\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") "
Sep 10 17:37:18.855305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855281    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c690d92c-6194-45ac-a3cd-b6aea0150967-tls-certs\") pod \"c690d92c-6194-45ac-a3cd-b6aea0150967\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") "
Sep 10 17:37:18.855516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855329    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-dshm\") pod \"c690d92c-6194-45ac-a3cd-b6aea0150967\" (UID: \"c690d92c-6194-45ac-a3cd-b6aea0150967\") "
Sep 10 17:37:18.855516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855405    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-model-cache" (OuterVolumeSpecName: "model-cache") pod "c690d92c-6194-45ac-a3cd-b6aea0150967" (UID: "c690d92c-6194-45ac-a3cd-b6aea0150967"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:18.855516 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855470    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-home" (OuterVolumeSpecName: "home") pod "c690d92c-6194-45ac-a3cd-b6aea0150967" (UID: "c690d92c-6194-45ac-a3cd-b6aea0150967"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:18.855998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855704    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:18.855998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855728    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:18.855998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.855952    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c690d92c-6194-45ac-a3cd-b6aea0150967" (UID: "c690d92c-6194-45ac-a3cd-b6aea0150967"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:18.857886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.857860    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c690d92c-6194-45ac-a3cd-b6aea0150967-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c690d92c-6194-45ac-a3cd-b6aea0150967" (UID: "c690d92c-6194-45ac-a3cd-b6aea0150967"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:37:18.858100 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.858078    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-dshm" (OuterVolumeSpecName: "dshm") pod "c690d92c-6194-45ac-a3cd-b6aea0150967" (UID: "c690d92c-6194-45ac-a3cd-b6aea0150967"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:37:18.858156 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.858095    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690d92c-6194-45ac-a3cd-b6aea0150967-kube-api-access-7kt7n" (OuterVolumeSpecName: "kube-api-access-7kt7n") pod "c690d92c-6194-45ac-a3cd-b6aea0150967" (UID: "c690d92c-6194-45ac-a3cd-b6aea0150967"). InnerVolumeSpecName "kube-api-access-7kt7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:18.956506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.956457    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kt7n\" (UniqueName: \"kubernetes.io/projected/c690d92c-6194-45ac-a3cd-b6aea0150967-kube-api-access-7kt7n\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:18.956506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.956490    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c690d92c-6194-45ac-a3cd-b6aea0150967-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:18.956506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.956501    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:18.956506 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:18.956508    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c690d92c-6194-45ac-a3cd-b6aea0150967-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:19.106382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.106343    2556 generic.go:356] "Generic (PLEG): container finished" podID="c690d92c-6194-45ac-a3cd-b6aea0150967" containerID="daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408" exitCode=0
Sep 10 17:37:19.106860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.106388    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"
Sep 10 17:37:19.106860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.106433    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw" event={"ID":"c690d92c-6194-45ac-a3cd-b6aea0150967","Type":"ContainerDied","Data":"daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408"}
Sep 10 17:37:19.106860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.106478    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw" event={"ID":"c690d92c-6194-45ac-a3cd-b6aea0150967","Type":"ContainerDied","Data":"184ad1faa9dd2646ee63bf4f04934fe095e208b2e270f98516c63427ff6724d1"}
Sep 10 17:37:19.106860 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.106500    2556 scope.go:117] "RemoveContainer" containerID="daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408"
Sep 10 17:37:19.116802 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.116777    2556 scope.go:117] "RemoveContainer" containerID="daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408"
Sep 10 17:37:19.117090 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:37:19.117068    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408\": container with ID starting with daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408 not found: ID does not exist" containerID="daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408"
Sep 10 17:37:19.117152 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.117105    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408"} err="failed to get container status \"daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408\": rpc error: code = NotFound desc = could not find container \"daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408\": container with ID starting with daee6d5224b7a126b08ad4a8c5fd4a4138374e3a84180334d7aa8afdfc1f5408 not found: ID does not exist"
Sep 10 17:37:19.133628 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.133589    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"]
Sep 10 17:37:19.136807 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.136730    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-6806f1/rate-test-model-6806f1-kserve-54589f5787-9rmtw"]
Sep 10 17:37:19.230027 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.230001    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:19.272033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.272002    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd0c05c-0b7d-4161-9550-fd09e75f7225" path="/var/lib/kubelet/pods/7cd0c05c-0b7d-4161-9550-fd09e75f7225/volumes"
Sep 10 17:37:19.272394 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.272381    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c690d92c-6194-45ac-a3cd-b6aea0150967" path="/var/lib/kubelet/pods/c690d92c-6194-45ac-a3cd-b6aea0150967/volumes"
Sep 10 17:37:19.360390 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.360297    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c36f87-d623-49d8-89de-10069981ba48-maas-api-service-ca\") pod \"29c36f87-d623-49d8-89de-10069981ba48\" (UID: \"29c36f87-d623-49d8-89de-10069981ba48\") "
Sep 10 17:37:19.360651 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.360628    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c36f87-d623-49d8-89de-10069981ba48-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "29c36f87-d623-49d8-89de-10069981ba48" (UID: "29c36f87-d623-49d8-89de-10069981ba48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:19.461511 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:19.461477    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29c36f87-d623-49d8-89de-10069981ba48-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:20.113556 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:20.113522    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd" event={"ID":"29c36f87-d623-49d8-89de-10069981ba48","Type":"ContainerDied","Data":"5b72486f5c947739e24d43cf2dc98b415b9ee6f63ff5734e1c4bfeaa89a54187"}
Sep 10 17:37:20.113962 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:20.113565    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"
Sep 10 17:37:20.113962 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:20.113571    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b72486f5c947739e24d43cf2dc98b415b9ee6f63ff5734e1c4bfeaa89a54187"
Sep 10 17:37:22.057426 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:22.057393    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:37:26.376814 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.376778    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"]
Sep 10 17:37:26.377259 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.377032    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7" podUID="3c41ae83-0b1c-4ed2-a346-34614f6fcfae" containerName="payload-processing" containerID="cri-o://0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7" gracePeriod=30
Sep 10 17:37:26.387454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.387424    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"]
Sep 10 17:37:26.387978 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.387940    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m" podUID="ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" containerName="payload-pre-processing" containerID="cri-o://3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a" gracePeriod=30
Sep 10 17:37:26.569984 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.569674    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"]
Sep 10 17:37:26.572104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.572073    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-edbcb5-ndrxd"]
Sep 10 17:37:26.673280 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.673258    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:37:26.675866 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.675847    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:37:26.728170 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728137    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-plugins-config-volume\") pod \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") "
Sep 10 17:37:26.728368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728230    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-plugins-config-volume\") pod \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") "
Sep 10 17:37:26.728368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728270    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs8jz\" (UniqueName: \"kubernetes.io/projected/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-kube-api-access-gs8jz\") pod \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\" (UID: \"3c41ae83-0b1c-4ed2-a346-34614f6fcfae\") "
Sep 10 17:37:26.728499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728367    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jwmz\" (UniqueName: \"kubernetes.io/projected/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-kube-api-access-5jwmz\") pod \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\" (UID: \"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d\") "
Sep 10 17:37:26.728559 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728537    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" (UID: "ad48b28e-bcac-460c-a6b2-4774ed4d3b9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:26.728684 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728636    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c41ae83-0b1c-4ed2-a346-34614f6fcfae" (UID: "3c41ae83-0b1c-4ed2-a346-34614f6fcfae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:37:26.728755 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.728680    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:26.730409 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.730386    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-kube-api-access-5jwmz" (OuterVolumeSpecName: "kube-api-access-5jwmz") pod "ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" (UID: "ad48b28e-bcac-460c-a6b2-4774ed4d3b9d"). InnerVolumeSpecName "kube-api-access-5jwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:26.730845 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.730825    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-kube-api-access-gs8jz" (OuterVolumeSpecName: "kube-api-access-gs8jz") pod "3c41ae83-0b1c-4ed2-a346-34614f6fcfae" (UID: "3c41ae83-0b1c-4ed2-a346-34614f6fcfae"). InnerVolumeSpecName "kube-api-access-gs8jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:37:26.829189 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.829149    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:26.829189 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.829184    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs8jz\" (UniqueName: \"kubernetes.io/projected/3c41ae83-0b1c-4ed2-a346-34614f6fcfae-kube-api-access-gs8jz\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:26.829189 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:26.829194    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jwmz\" (UniqueName: \"kubernetes.io/projected/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d-kube-api-access-5jwmz\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:37:27.154229 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.154181    2556 generic.go:356] "Generic (PLEG): container finished" podID="3c41ae83-0b1c-4ed2-a346-34614f6fcfae" containerID="0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7" exitCode=0
Sep 10 17:37:27.154229 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.154227    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"
Sep 10 17:37:27.154499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.154259    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7" event={"ID":"3c41ae83-0b1c-4ed2-a346-34614f6fcfae","Type":"ContainerDied","Data":"0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7"}
Sep 10 17:37:27.154499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.154324    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7" event={"ID":"3c41ae83-0b1c-4ed2-a346-34614f6fcfae","Type":"ContainerDied","Data":"b18a72c168b1f9ca6b8fe762cb49b7b0c858980cdc399bc1e41ce5a3d876d3e8"}
Sep 10 17:37:27.154499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.154347    2556 scope.go:117] "RemoveContainer" containerID="0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7"
Sep 10 17:37:27.155884 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.155857    2556 generic.go:356] "Generic (PLEG): container finished" podID="ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" containerID="3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a" exitCode=0
Sep 10 17:37:27.156021 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.155892    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"
Sep 10 17:37:27.156021 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.155946    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m" event={"ID":"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d","Type":"ContainerDied","Data":"3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a"}
Sep 10 17:37:27.156021 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.155990    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m" event={"ID":"ad48b28e-bcac-460c-a6b2-4774ed4d3b9d","Type":"ContainerDied","Data":"27cb5386006a9cf0f7f776e3b60877d1834f3b5631c1bb13ea4539f6534d9d70"}
Sep 10 17:37:27.168318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.165764    2556 scope.go:117] "RemoveContainer" containerID="0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7"
Sep 10 17:37:27.168467 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:37:27.168307    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7\": container with ID starting with 0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7 not found: ID does not exist" containerID="0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7"
Sep 10 17:37:27.168467 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.168338    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7"} err="failed to get container status \"0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7\": rpc error: code = NotFound desc = could not find container \"0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7\": container with ID starting with 0a95c722f07ad884f81d4630984324b1634769ffe78ed1fbaa1c4baaa3c8f0b7 not found: ID does not exist"
Sep 10 17:37:27.168467 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.168360    2556 scope.go:117] "RemoveContainer" containerID="3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a"
Sep 10 17:37:27.177931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.177910    2556 scope.go:117] "RemoveContainer" containerID="3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a"
Sep 10 17:37:27.178199 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:37:27.178179    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a\": container with ID starting with 3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a not found: ID does not exist" containerID="3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a"
Sep 10 17:37:27.178285 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.178210    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a"} err="failed to get container status \"3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a\": rpc error: code = NotFound desc = could not find container \"3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a\": container with ID starting with 3febd7039a1d36338112aa680fe34065804f6d29a5438b3eb2f9be725b8eaf1a not found: ID does not exist"
Sep 10 17:37:27.180839 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.180813    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"]
Sep 10 17:37:27.187592 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.187572    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-edbcb5-5d6f76645c-4h7s7"]
Sep 10 17:37:27.199345 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.199323    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"]
Sep 10 17:37:27.203568 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.203547    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-edbcb5-5796c7f67d-8t92m"]
Sep 10 17:37:27.271967 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.271931    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c36f87-d623-49d8-89de-10069981ba48" path="/var/lib/kubelet/pods/29c36f87-d623-49d8-89de-10069981ba48/volumes"
Sep 10 17:37:27.272315 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.272301    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c41ae83-0b1c-4ed2-a346-34614f6fcfae" path="/var/lib/kubelet/pods/3c41ae83-0b1c-4ed2-a346-34614f6fcfae/volumes"
Sep 10 17:37:27.272619 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:27.272607    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" path="/var/lib/kubelet/pods/ad48b28e-bcac-460c-a6b2-4774ed4d3b9d/volumes"
Sep 10 17:37:38.510996 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.510958    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"]
Sep 10 17:37:38.511377 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511350    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c36f87-d623-49d8-89de-10069981ba48" containerName="revoke-keys"
Sep 10 17:37:38.511377 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511361    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c36f87-d623-49d8-89de-10069981ba48" containerName="revoke-keys"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511383    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" containerName="payload-pre-processing"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511390    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" containerName="payload-pre-processing"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511398    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c690d92c-6194-45ac-a3cd-b6aea0150967" containerName="main"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511404    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c690d92c-6194-45ac-a3cd-b6aea0150967" containerName="main"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511415    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd0c05c-0b7d-4161-9550-fd09e75f7225" containerName="main"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511421    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd0c05c-0b7d-4161-9550-fd09e75f7225" containerName="main"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511432    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c41ae83-0b1c-4ed2-a346-34614f6fcfae" containerName="payload-processing"
Sep 10 17:37:38.511454 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511437    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c41ae83-0b1c-4ed2-a346-34614f6fcfae" containerName="payload-processing"
Sep 10 17:37:38.511733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511495    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd0c05c-0b7d-4161-9550-fd09e75f7225" containerName="main"
Sep 10 17:37:38.511733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511504    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c690d92c-6194-45ac-a3cd-b6aea0150967" containerName="main"
Sep 10 17:37:38.511733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511511    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad48b28e-bcac-460c-a6b2-4774ed4d3b9d" containerName="payload-pre-processing"
Sep 10 17:37:38.511733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511518    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c36f87-d623-49d8-89de-10069981ba48" containerName="revoke-keys"
Sep 10 17:37:38.511733 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.511524    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c41ae83-0b1c-4ed2-a346-34614f6fcfae" containerName="payload-processing"
Sep 10 17:37:38.513658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.513641    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.516048 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.516027    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-777ae3\""
Sep 10 17:37:38.516617 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.516594    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-777ae3-dockercfg-hj8dd\""
Sep 10 17:37:38.523721 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.523650    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"]
Sep 10 17:37:38.553971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.553943    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"]
Sep 10 17:37:38.556579 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.556560    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.570400 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.570372    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"]
Sep 10 17:37:38.643457 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.643421    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb0890-2b61-408d-b7b0-4900546b8e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.643627 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.643478    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjgv\" (UniqueName: \"kubernetes.io/projected/e5eb0890-2b61-408d-b7b0-4900546b8e88-kube-api-access-sfjgv\") pod \"payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.643627 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.643559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.643627 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.643609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2dr\" (UniqueName: \"kubernetes.io/projected/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-kube-api-access-vk2dr\") pod \"payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.744790 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.744752    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2dr\" (UniqueName: \"kubernetes.io/projected/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-kube-api-access-vk2dr\") pod \"payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.744974 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.744826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb0890-2b61-408d-b7b0-4900546b8e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.744974 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.744867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjgv\" (UniqueName: \"kubernetes.io/projected/e5eb0890-2b61-408d-b7b0-4900546b8e88-kube-api-access-sfjgv\") pod \"payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.744974 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.744929    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.745634 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.745613    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb0890-2b61-408d-b7b0-4900546b8e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.745699 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.745626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.756115 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.756082    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjgv\" (UniqueName: \"kubernetes.io/projected/e5eb0890-2b61-408d-b7b0-4900546b8e88-kube-api-access-sfjgv\") pod \"payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:38.756235 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.756082    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2dr\" (UniqueName: \"kubernetes.io/projected/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-kube-api-access-vk2dr\") pod \"payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.824954 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.824866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:38.866155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:38.866119    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:39.100748 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.100644    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"]
Sep 10 17:37:39.111075 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:37:39.111049    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69efec71_60e0_428b_9bc1_faf6e4d2cb0c.slice/crio-46734d179f2cf5652a691fbffbb9b44f7a1aa7b9c4cb32044aaad369ca2417d9 WatchSource:0}: Error finding container 46734d179f2cf5652a691fbffbb9b44f7a1aa7b9c4cb32044aaad369ca2417d9: Status 404 returned error can't find the container with id 46734d179f2cf5652a691fbffbb9b44f7a1aa7b9c4cb32044aaad369ca2417d9
Sep 10 17:37:39.148412 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.148386    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"]
Sep 10 17:37:39.151297 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:37:39.151265    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5eb0890_2b61_408d_b7b0_4900546b8e88.slice/crio-0a2d05eea6c326e4efb10cc6be8b420ef5ab085cbd49ae6123134e2f22479812 WatchSource:0}: Error finding container 0a2d05eea6c326e4efb10cc6be8b420ef5ab085cbd49ae6123134e2f22479812: Status 404 returned error can't find the container with id 0a2d05eea6c326e4efb10cc6be8b420ef5ab085cbd49ae6123134e2f22479812
Sep 10 17:37:39.226111 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.226066    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg" event={"ID":"e5eb0890-2b61-408d-b7b0-4900546b8e88","Type":"ContainerStarted","Data":"975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970"}
Sep 10 17:37:39.226307 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.226136    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:39.226307 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.226165    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg" event={"ID":"e5eb0890-2b61-408d-b7b0-4900546b8e88","Type":"ContainerStarted","Data":"0a2d05eea6c326e4efb10cc6be8b420ef5ab085cbd49ae6123134e2f22479812"}
Sep 10 17:37:39.227869 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.227842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw" event={"ID":"69efec71-60e0-428b-9bc1-faf6e4d2cb0c","Type":"ContainerStarted","Data":"46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151"}
Sep 10 17:37:39.228001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.227889    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw" event={"ID":"69efec71-60e0-428b-9bc1-faf6e4d2cb0c","Type":"ContainerStarted","Data":"46734d179f2cf5652a691fbffbb9b44f7a1aa7b9c4cb32044aaad369ca2417d9"}
Sep 10 17:37:39.228001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.227986    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:37:39.249289 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.249217    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg" podStartSLOduration=1.249197219 podStartE2EDuration="1.249197219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:39.247013803 +0000 UTC m=+1544.467137737" watchObservedRunningTime="2026-09-10 17:37:39.249197219 +0000 UTC m=+1544.469321155"
Sep 10 17:37:39.271299 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:39.271229    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw" podStartSLOduration=1.2712084639999999 podStartE2EDuration="1.271208464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:37:39.268735244 +0000 UTC m=+1544.488859177" watchObservedRunningTime="2026-09-10 17:37:39.271208464 +0000 UTC m=+1544.491332397"
Sep 10 17:37:50.235167 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:50.235135    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:37:50.235627 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:37:50.235210    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:38:06.605162 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:06.605126    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wjvg7_97929e09-8ffb-4a86-b440-b2bed7e9a3ac/payload-processing/0.log"
Sep 10 17:38:06.720033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:06.720001    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw_69efec71-60e0-428b-9bc1-faf6e4d2cb0c/payload-processing/0.log"
Sep 10 17:38:23.926323 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.926286    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"]
Sep 10 17:38:23.930305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.930281    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:23.933969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.933943    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-437dd7\"/\"default-dockercfg-g4r59\""
Sep 10 17:38:23.934104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.934030    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-437dd7\"/\"ipp-route-437dd7-kserve-self-signed-certs\""
Sep 10 17:38:23.934179 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.934106    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-437dd7\"/\"kube-root-ca.crt\""
Sep 10 17:38:23.934179 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.934170    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-437dd7\"/\"openshift-service-ca.crt\""
Sep 10 17:38:23.941729 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:23.941707    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"]
Sep 10 17:38:24.085846 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.085805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tls-certs\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.085846 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.085847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-dshm\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.086097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.085871    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8b8f\" (UniqueName: \"kubernetes.io/projected/d17a5530-ebd6-4583-b15a-591d2ce38bbc-kube-api-access-n8b8f\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.086097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.085895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-home\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.086097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.085961    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-model-cache\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.086097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.085990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tmp-dir\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187488 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187387    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-model-cache\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187488 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tmp-dir\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187777 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tls-certs\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187777 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187579    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-dshm\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187777 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8b8f\" (UniqueName: \"kubernetes.io/projected/d17a5530-ebd6-4583-b15a-591d2ce38bbc-kube-api-access-n8b8f\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187777 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187640    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-home\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187879    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-model-cache\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.187987 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.187909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tmp-dir\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.188100 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.188070    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-home\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.189939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.189912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-dshm\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.190084 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.190066    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tls-certs\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.195905 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.195882    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8b8f\" (UniqueName: \"kubernetes.io/projected/d17a5530-ebd6-4583-b15a-591d2ce38bbc-kube-api-access-n8b8f\") pod \"ipp-route-437dd7-kserve-6d45684cff-tq69m\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") " pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.241601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.241559    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:24.521073 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:24.520264    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"]
Sep 10 17:38:24.523361 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:38:24.523235    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd17a5530_ebd6_4583_b15a_591d2ce38bbc.slice/crio-5661f304776c732d96edb14cf444983779cbd69b1ff8e2318dc60f09f0fb25ca WatchSource:0}: Error finding container 5661f304776c732d96edb14cf444983779cbd69b1ff8e2318dc60f09f0fb25ca: Status 404 returned error can't find the container with id 5661f304776c732d96edb14cf444983779cbd69b1ff8e2318dc60f09f0fb25ca
Sep 10 17:38:25.484105 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:25.484067    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m" event={"ID":"d17a5530-ebd6-4583-b15a-591d2ce38bbc","Type":"ContainerStarted","Data":"2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf"}
Sep 10 17:38:25.484630 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:25.484115    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m" event={"ID":"d17a5530-ebd6-4583-b15a-591d2ce38bbc","Type":"ContainerStarted","Data":"5661f304776c732d96edb14cf444983779cbd69b1ff8e2318dc60f09f0fb25ca"}
Sep 10 17:38:25.505683 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:25.505609    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m" podStartSLOduration=2.505595113 podStartE2EDuration="2.505595113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:25.503901645 +0000 UTC m=+1590.724025570" watchObservedRunningTime="2026-09-10 17:38:25.505595113 +0000 UTC m=+1590.725719045"
Sep 10 17:38:34.242601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:34.242555    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:34.242601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:34.242606    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:34.255572 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:34.255544    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:34.549562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:34.549474    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:38:52.553081 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:52.553013    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs_5ed3f6bb-2d1e-406e-990f-5a674180372f/payload-processing/0.log"
Sep 10 17:38:52.672563 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:52.672535    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wjvg7_97929e09-8ffb-4a86-b440-b2bed7e9a3ac/payload-processing/0.log"
Sep 10 17:38:52.792637 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:52.792609    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw_69efec71-60e0-428b-9bc1-faf6e4d2cb0c/payload-processing/0.log"
Sep 10 17:38:55.482233 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:55.482201    2556 scope.go:117] "RemoveContainer" containerID="c1d521b7afc0001cd27a33177c0e810ce45243f089f473f7dd1150906132422c"
Sep 10 17:38:58.074754 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.074720    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"]
Sep 10 17:38:58.078062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.078046    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.080932 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.080901    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5eebb3\""
Sep 10 17:38:58.081114 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.080946    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5eebb3-dockercfg-8d6q6\""
Sep 10 17:38:58.089962 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.089939    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"]
Sep 10 17:38:58.092155 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.092132    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"]
Sep 10 17:38:58.095939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.095919    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.108275 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.108254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"]
Sep 10 17:38:58.214753 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.214706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb97z\" (UniqueName: \"kubernetes.io/projected/1be00aa2-7167-462f-8ff5-b964026f93c6-kube-api-access-qb97z\") pod \"payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.214958 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.214866    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwqd\" (UniqueName: \"kubernetes.io/projected/01bf8287-578b-4b6d-af16-dad454b3cd93-kube-api-access-xpwqd\") pod \"payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.214958 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.214911    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be00aa2-7167-462f-8ff5-b964026f93c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.214958 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.214935    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bf8287-578b-4b6d-af16-dad454b3cd93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.316353 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.316317    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwqd\" (UniqueName: \"kubernetes.io/projected/01bf8287-578b-4b6d-af16-dad454b3cd93-kube-api-access-xpwqd\") pod \"payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.316353 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.316360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be00aa2-7167-462f-8ff5-b964026f93c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.316603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.316378    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bf8287-578b-4b6d-af16-dad454b3cd93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.316603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.316511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb97z\" (UniqueName: \"kubernetes.io/projected/1be00aa2-7167-462f-8ff5-b964026f93c6-kube-api-access-qb97z\") pod \"payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.317363 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.317337    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be00aa2-7167-462f-8ff5-b964026f93c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.317562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.317503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bf8287-578b-4b6d-af16-dad454b3cd93-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.324633 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.324605    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb97z\" (UniqueName: \"kubernetes.io/projected/1be00aa2-7167-462f-8ff5-b964026f93c6-kube-api-access-qb97z\") pod \"payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.324777 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.324734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwqd\" (UniqueName: \"kubernetes.io/projected/01bf8287-578b-4b6d-af16-dad454b3cd93-kube-api-access-xpwqd\") pod \"payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.389648 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.389609    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:58.405990 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.405958    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:58.667771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.667715    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"]
Sep 10 17:38:58.669995 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:38:58.669967    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1be00aa2_7167_462f_8ff5_b964026f93c6.slice/crio-28917b98b2301d31f49b7a12c6bfccafc1dccf1a10d92a8f5a75a3d57a75e3f3 WatchSource:0}: Error finding container 28917b98b2301d31f49b7a12c6bfccafc1dccf1a10d92a8f5a75a3d57a75e3f3: Status 404 returned error can't find the container with id 28917b98b2301d31f49b7a12c6bfccafc1dccf1a10d92a8f5a75a3d57a75e3f3
Sep 10 17:38:58.703965 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:58.703936    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"]
Sep 10 17:38:58.705063 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:38:58.705040    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01bf8287_578b_4b6d_af16_dad454b3cd93.slice/crio-b066fdf4db06ae9c4e3516678bd91fd03d3c01d78746589e228c42bb8d199636 WatchSource:0}: Error finding container b066fdf4db06ae9c4e3516678bd91fd03d3c01d78746589e228c42bb8d199636: Status 404 returned error can't find the container with id b066fdf4db06ae9c4e3516678bd91fd03d3c01d78746589e228c42bb8d199636
Sep 10 17:38:59.667612 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.667561    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls" event={"ID":"01bf8287-578b-4b6d-af16-dad454b3cd93","Type":"ContainerStarted","Data":"aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b"}
Sep 10 17:38:59.668102 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.667627    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:38:59.668102 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.667651    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls" event={"ID":"01bf8287-578b-4b6d-af16-dad454b3cd93","Type":"ContainerStarted","Data":"b066fdf4db06ae9c4e3516678bd91fd03d3c01d78746589e228c42bb8d199636"}
Sep 10 17:38:59.669303 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.669275    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f" event={"ID":"1be00aa2-7167-462f-8ff5-b964026f93c6","Type":"ContainerStarted","Data":"b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715"}
Sep 10 17:38:59.669431 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.669326    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:38:59.669431 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.669354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f" event={"ID":"1be00aa2-7167-462f-8ff5-b964026f93c6","Type":"ContainerStarted","Data":"28917b98b2301d31f49b7a12c6bfccafc1dccf1a10d92a8f5a75a3d57a75e3f3"}
Sep 10 17:38:59.687537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.687489    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls" podStartSLOduration=1.687475113 podStartE2EDuration="1.687475113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:59.684202979 +0000 UTC m=+1624.904326912" watchObservedRunningTime="2026-09-10 17:38:59.687475113 +0000 UTC m=+1624.907599047"
Sep 10 17:38:59.703766 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:38:59.703721    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f" podStartSLOduration=1.7037108920000001 podStartE2EDuration="1.703710892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:38:59.702450612 +0000 UTC m=+1624.922574545" watchObservedRunningTime="2026-09-10 17:38:59.703710892 +0000 UTC m=+1624.923834826"
Sep 10 17:39:10.677683 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:10.677635    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:39:10.678147 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:10.677730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:39:21.719679 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.719633    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"]
Sep 10 17:39:21.723492 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.723476    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:21.726043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.726022    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:39:21.726128 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.726024    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:39:21.730656 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.730632    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"]
Sep 10 17:39:21.846695 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.846634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b01bb9-189b-4e04-98d4-c2ba07b861de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv\" (UID: \"70b01bb9-189b-4e04-98d4-c2ba07b861de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:21.947684 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.947631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b01bb9-189b-4e04-98d4-c2ba07b861de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv\" (UID: \"70b01bb9-189b-4e04-98d4-c2ba07b861de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:21.948295 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:21.948265    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b01bb9-189b-4e04-98d4-c2ba07b861de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv\" (UID: \"70b01bb9-189b-4e04-98d4-c2ba07b861de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:22.035185 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:22.035086    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:22.307175 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:22.307148    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"]
Sep 10 17:39:22.309094 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:39:22.309064    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b01bb9_189b_4e04_98d4_c2ba07b861de.slice/crio-974f8730ea30f00dea2570aa9938c14da68aaec7b95804262e9dfa29f349ce98 WatchSource:0}: Error finding container 974f8730ea30f00dea2570aa9938c14da68aaec7b95804262e9dfa29f349ce98: Status 404 returned error can't find the container with id 974f8730ea30f00dea2570aa9938c14da68aaec7b95804262e9dfa29f349ce98
Sep 10 17:39:22.797702 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:22.797655    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv" event={"ID":"70b01bb9-189b-4e04-98d4-c2ba07b861de","Type":"ContainerStarted","Data":"33a61b35bcfd0106ff0ea8cdfd01253dff1d5b68ef241358e1cf7e101c775434"}
Sep 10 17:39:22.798120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:22.797727    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv" event={"ID":"70b01bb9-189b-4e04-98d4-c2ba07b861de","Type":"ContainerStarted","Data":"974f8730ea30f00dea2570aa9938c14da68aaec7b95804262e9dfa29f349ce98"}
Sep 10 17:39:22.813071 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:22.813018    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv" podStartSLOduration=1.8130004469999998 podStartE2EDuration="1.813000447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:39:22.812814177 +0000 UTC m=+1648.032938110" watchObservedRunningTime="2026-09-10 17:39:22.813000447 +0000 UTC m=+1648.033124379"
Sep 10 17:39:23.806059 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:23.806022    2556 generic.go:356] "Generic (PLEG): container finished" podID="70b01bb9-189b-4e04-98d4-c2ba07b861de" containerID="33a61b35bcfd0106ff0ea8cdfd01253dff1d5b68ef241358e1cf7e101c775434" exitCode=0
Sep 10 17:39:23.806429 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:23.806123    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv" event={"ID":"70b01bb9-189b-4e04-98d4-c2ba07b861de","Type":"ContainerDied","Data":"33a61b35bcfd0106ff0ea8cdfd01253dff1d5b68ef241358e1cf7e101c775434"}
Sep 10 17:39:24.955809 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:24.955782    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:25.081169 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:25.081082    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b01bb9-189b-4e04-98d4-c2ba07b861de-maas-api-service-ca\") pod \"70b01bb9-189b-4e04-98d4-c2ba07b861de\" (UID: \"70b01bb9-189b-4e04-98d4-c2ba07b861de\") "
Sep 10 17:39:25.081448 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:25.081425    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b01bb9-189b-4e04-98d4-c2ba07b861de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70b01bb9-189b-4e04-98d4-c2ba07b861de" (UID: "70b01bb9-189b-4e04-98d4-c2ba07b861de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:25.182812 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:25.182778    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b01bb9-189b-4e04-98d4-c2ba07b861de-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:25.824510 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:25.824473    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"
Sep 10 17:39:25.824756 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:25.824510    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv" event={"ID":"70b01bb9-189b-4e04-98d4-c2ba07b861de","Type":"ContainerDied","Data":"974f8730ea30f00dea2570aa9938c14da68aaec7b95804262e9dfa29f349ce98"}
Sep 10 17:39:25.824756 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:25.824571    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="974f8730ea30f00dea2570aa9938c14da68aaec7b95804262e9dfa29f349ce98"
Sep 10 17:39:27.977254 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:27.977210    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"]
Sep 10 17:39:27.977859 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:27.977452    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls" podUID="01bf8287-578b-4b6d-af16-dad454b3cd93" containerName="payload-processing" containerID="cri-o://aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b" gracePeriod=30
Sep 10 17:39:27.985717 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:27.985692    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"]
Sep 10 17:39:27.985948 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:27.985920    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f" podUID="1be00aa2-7167-462f-8ff5-b964026f93c6" containerName="payload-pre-processing" containerID="cri-o://b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715" gracePeriod=30
Sep 10 17:39:28.228789 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.228701    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"]
Sep 10 17:39:28.231810 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.231784    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-5eebb3-qkdjv"]
Sep 10 17:39:28.267845 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.267817    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:39:28.270731 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.270712    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:39:28.414943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.414906    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bf8287-578b-4b6d-af16-dad454b3cd93-plugins-config-volume\") pod \"01bf8287-578b-4b6d-af16-dad454b3cd93\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") "
Sep 10 17:39:28.415125 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.415048    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb97z\" (UniqueName: \"kubernetes.io/projected/1be00aa2-7167-462f-8ff5-b964026f93c6-kube-api-access-qb97z\") pod \"1be00aa2-7167-462f-8ff5-b964026f93c6\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") "
Sep 10 17:39:28.415125 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.415106    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpwqd\" (UniqueName: \"kubernetes.io/projected/01bf8287-578b-4b6d-af16-dad454b3cd93-kube-api-access-xpwqd\") pod \"01bf8287-578b-4b6d-af16-dad454b3cd93\" (UID: \"01bf8287-578b-4b6d-af16-dad454b3cd93\") "
Sep 10 17:39:28.415216 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.415149    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be00aa2-7167-462f-8ff5-b964026f93c6-plugins-config-volume\") pod \"1be00aa2-7167-462f-8ff5-b964026f93c6\" (UID: \"1be00aa2-7167-462f-8ff5-b964026f93c6\") "
Sep 10 17:39:28.415367 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.415347    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01bf8287-578b-4b6d-af16-dad454b3cd93-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01bf8287-578b-4b6d-af16-dad454b3cd93" (UID: "01bf8287-578b-4b6d-af16-dad454b3cd93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:28.415494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.415478    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01bf8287-578b-4b6d-af16-dad454b3cd93-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:28.415600 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.415577    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1be00aa2-7167-462f-8ff5-b964026f93c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1be00aa2-7167-462f-8ff5-b964026f93c6" (UID: "1be00aa2-7167-462f-8ff5-b964026f93c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:28.417220 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.417200    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1be00aa2-7167-462f-8ff5-b964026f93c6-kube-api-access-qb97z" (OuterVolumeSpecName: "kube-api-access-qb97z") pod "1be00aa2-7167-462f-8ff5-b964026f93c6" (UID: "1be00aa2-7167-462f-8ff5-b964026f93c6"). InnerVolumeSpecName "kube-api-access-qb97z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:39:28.417290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.417218    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01bf8287-578b-4b6d-af16-dad454b3cd93-kube-api-access-xpwqd" (OuterVolumeSpecName: "kube-api-access-xpwqd") pod "01bf8287-578b-4b6d-af16-dad454b3cd93" (UID: "01bf8287-578b-4b6d-af16-dad454b3cd93"). InnerVolumeSpecName "kube-api-access-xpwqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:39:28.516955 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.516864    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb97z\" (UniqueName: \"kubernetes.io/projected/1be00aa2-7167-462f-8ff5-b964026f93c6-kube-api-access-qb97z\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:28.516955 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.516894    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwqd\" (UniqueName: \"kubernetes.io/projected/01bf8287-578b-4b6d-af16-dad454b3cd93-kube-api-access-xpwqd\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:28.516955 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.516904    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1be00aa2-7167-462f-8ff5-b964026f93c6-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:28.843193 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.843065    2556 generic.go:356] "Generic (PLEG): container finished" podID="01bf8287-578b-4b6d-af16-dad454b3cd93" containerID="aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b" exitCode=0
Sep 10 17:39:28.843193 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.843113    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"
Sep 10 17:39:28.843193 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.843150    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls" event={"ID":"01bf8287-578b-4b6d-af16-dad454b3cd93","Type":"ContainerDied","Data":"aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b"}
Sep 10 17:39:28.843521 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.843211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls" event={"ID":"01bf8287-578b-4b6d-af16-dad454b3cd93","Type":"ContainerDied","Data":"b066fdf4db06ae9c4e3516678bd91fd03d3c01d78746589e228c42bb8d199636"}
Sep 10 17:39:28.843521 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.843245    2556 scope.go:117] "RemoveContainer" containerID="aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b"
Sep 10 17:39:28.845213 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.845184    2556 generic.go:356] "Generic (PLEG): container finished" podID="1be00aa2-7167-462f-8ff5-b964026f93c6" containerID="b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715" exitCode=0
Sep 10 17:39:28.845353 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.845224    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"
Sep 10 17:39:28.845353 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.845284    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f" event={"ID":"1be00aa2-7167-462f-8ff5-b964026f93c6","Type":"ContainerDied","Data":"b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715"}
Sep 10 17:39:28.845353 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.845325    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f" event={"ID":"1be00aa2-7167-462f-8ff5-b964026f93c6","Type":"ContainerDied","Data":"28917b98b2301d31f49b7a12c6bfccafc1dccf1a10d92a8f5a75a3d57a75e3f3"}
Sep 10 17:39:28.854060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.854041    2556 scope.go:117] "RemoveContainer" containerID="aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b"
Sep 10 17:39:28.854427 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:39:28.854399    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b\": container with ID starting with aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b not found: ID does not exist" containerID="aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b"
Sep 10 17:39:28.854519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.854436    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b"} err="failed to get container status \"aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b\": rpc error: code = NotFound desc = could not find container \"aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b\": container with ID starting with aa1c9ab05236879dc0320e40a8dde0ac187369959c138ab9e50aff14aebae81b not found: ID does not exist"
Sep 10 17:39:28.854519 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.854455    2556 scope.go:117] "RemoveContainer" containerID="b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715"
Sep 10 17:39:28.863062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.863044    2556 scope.go:117] "RemoveContainer" containerID="b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715"
Sep 10 17:39:28.863319 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:39:28.863304    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715\": container with ID starting with b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715 not found: ID does not exist" containerID="b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715"
Sep 10 17:39:28.863366 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.863326    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715"} err="failed to get container status \"b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715\": rpc error: code = NotFound desc = could not find container \"b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715\": container with ID starting with b44d5de1cd2d0e73ee591ce1b87d134080911f6c792db41fa27b8403c7499715 not found: ID does not exist"
Sep 10 17:39:28.868509 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.868485    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"]
Sep 10 17:39:28.872832 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.872809    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5eebb3-6bb68b68c9-9f4ls"]
Sep 10 17:39:28.886950 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.886920    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"]
Sep 10 17:39:28.893012 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:28.892986    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5eebb3-7c46ff7f5d-d558f"]
Sep 10 17:39:29.273029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:29.272992    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01bf8287-578b-4b6d-af16-dad454b3cd93" path="/var/lib/kubelet/pods/01bf8287-578b-4b6d-af16-dad454b3cd93/volumes"
Sep 10 17:39:29.273396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:29.273375    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1be00aa2-7167-462f-8ff5-b964026f93c6" path="/var/lib/kubelet/pods/1be00aa2-7167-462f-8ff5-b964026f93c6/volumes"
Sep 10 17:39:29.273690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:29.273654    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b01bb9-189b-4e04-98d4-c2ba07b861de" path="/var/lib/kubelet/pods/70b01bb9-189b-4e04-98d4-c2ba07b861de/volumes"
Sep 10 17:39:39.085932 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.085898    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"]
Sep 10 17:39:39.086351 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086321    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01bf8287-578b-4b6d-af16-dad454b3cd93" containerName="payload-processing"
Sep 10 17:39:39.086351 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086332    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="01bf8287-578b-4b6d-af16-dad454b3cd93" containerName="payload-processing"
Sep 10 17:39:39.086443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086354    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1be00aa2-7167-462f-8ff5-b964026f93c6" containerName="payload-pre-processing"
Sep 10 17:39:39.086443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086360    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1be00aa2-7167-462f-8ff5-b964026f93c6" containerName="payload-pre-processing"
Sep 10 17:39:39.086443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086369    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b01bb9-189b-4e04-98d4-c2ba07b861de" containerName="revoke-keys"
Sep 10 17:39:39.086443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086375    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b01bb9-189b-4e04-98d4-c2ba07b861de" containerName="revoke-keys"
Sep 10 17:39:39.086443 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086436    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b01bb9-189b-4e04-98d4-c2ba07b861de" containerName="revoke-keys"
Sep 10 17:39:39.086622 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086446    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1be00aa2-7167-462f-8ff5-b964026f93c6" containerName="payload-pre-processing"
Sep 10 17:39:39.086622 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.086452    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="01bf8287-578b-4b6d-af16-dad454b3cd93" containerName="payload-processing"
Sep 10 17:39:39.090979 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.090956    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:39.094167 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.094144    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:39:39.094939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.094921    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:39:39.102783 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.102764    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"]
Sep 10 17:39:39.224248 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.224210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2f9ee8-f70f-42b5-adf9-51a7083f4140-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll\" (UID: \"3a2f9ee8-f70f-42b5-adf9-51a7083f4140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:39.325822 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.325788    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2f9ee8-f70f-42b5-adf9-51a7083f4140-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll\" (UID: \"3a2f9ee8-f70f-42b5-adf9-51a7083f4140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:39.326382 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.326361    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2f9ee8-f70f-42b5-adf9-51a7083f4140-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll\" (UID: \"3a2f9ee8-f70f-42b5-adf9-51a7083f4140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:39.402029 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.401949    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:39.680091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.680005    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"]
Sep 10 17:39:39.683316 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:39:39.683290    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2f9ee8_f70f_42b5_adf9_51a7083f4140.slice/crio-339bafa90454fd795868c8d0ad1f9255ccab53bb05044db0598bdb0109913242 WatchSource:0}: Error finding container 339bafa90454fd795868c8d0ad1f9255ccab53bb05044db0598bdb0109913242: Status 404 returned error can't find the container with id 339bafa90454fd795868c8d0ad1f9255ccab53bb05044db0598bdb0109913242
Sep 10 17:39:39.913729 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:39.913658    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll" event={"ID":"3a2f9ee8-f70f-42b5-adf9-51a7083f4140","Type":"ContainerStarted","Data":"339bafa90454fd795868c8d0ad1f9255ccab53bb05044db0598bdb0109913242"}
Sep 10 17:39:40.921250 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:40.921204    2556 generic.go:356] "Generic (PLEG): container finished" podID="3a2f9ee8-f70f-42b5-adf9-51a7083f4140" containerID="d4992f305dd94534e29639d1a0c0d4b1a606e7f81f2a46588cd94b56489ee1dc" exitCode=0
Sep 10 17:39:40.921649 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:40.921288    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll" event={"ID":"3a2f9ee8-f70f-42b5-adf9-51a7083f4140","Type":"ContainerDied","Data":"d4992f305dd94534e29639d1a0c0d4b1a606e7f81f2a46588cd94b56489ee1dc"}
Sep 10 17:39:42.062317 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.062292    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:42.257086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.257055    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2f9ee8-f70f-42b5-adf9-51a7083f4140-maas-api-service-ca\") pod \"3a2f9ee8-f70f-42b5-adf9-51a7083f4140\" (UID: \"3a2f9ee8-f70f-42b5-adf9-51a7083f4140\") "
Sep 10 17:39:42.257416 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.257394    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a2f9ee8-f70f-42b5-adf9-51a7083f4140-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a2f9ee8-f70f-42b5-adf9-51a7083f4140" (UID: "3a2f9ee8-f70f-42b5-adf9-51a7083f4140"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:42.358246 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.358217    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a2f9ee8-f70f-42b5-adf9-51a7083f4140-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:42.933434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.933393    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"
Sep 10 17:39:42.933434 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.933426    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll" event={"ID":"3a2f9ee8-f70f-42b5-adf9-51a7083f4140","Type":"ContainerDied","Data":"339bafa90454fd795868c8d0ad1f9255ccab53bb05044db0598bdb0109913242"}
Sep 10 17:39:42.933687 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:42.933481    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="339bafa90454fd795868c8d0ad1f9255ccab53bb05044db0598bdb0109913242"
Sep 10 17:39:54.727522 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:54.727486    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"]
Sep 10 17:39:54.727983 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:54.727732    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs" podUID="5ed3f6bb-2d1e-406e-990f-5a674180372f" containerName="payload-processing" containerID="cri-o://e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4" gracePeriod=30
Sep 10 17:39:54.984985 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:54.984928    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:39:55.001753 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.001725    2556 generic.go:356] "Generic (PLEG): container finished" podID="5ed3f6bb-2d1e-406e-990f-5a674180372f" containerID="e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4" exitCode=0
Sep 10 17:39:55.001887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.001778    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"
Sep 10 17:39:55.001887 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.001820    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs" event={"ID":"5ed3f6bb-2d1e-406e-990f-5a674180372f","Type":"ContainerDied","Data":"e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4"}
Sep 10 17:39:55.001975 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.001913    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs" event={"ID":"5ed3f6bb-2d1e-406e-990f-5a674180372f","Type":"ContainerDied","Data":"1ffae58af8c909283b293d9737139b1c5e3e85886794cad8ccd49ebb2ceb06bf"}
Sep 10 17:39:55.001975 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.001946    2556 scope.go:117] "RemoveContainer" containerID="e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4"
Sep 10 17:39:55.011525 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.011286    2556 scope.go:117] "RemoveContainer" containerID="e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4"
Sep 10 17:39:55.012022 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:39:55.011986    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4\": container with ID starting with e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4 not found: ID does not exist" containerID="e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4"
Sep 10 17:39:55.012431 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.012021    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4"} err="failed to get container status \"e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4\": rpc error: code = NotFound desc = could not find container \"e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4\": container with ID starting with e77635e2a8f88c29854d76bb37c7774dfdaf3d736a354dd02fbaa9f08f659be4 not found: ID does not exist"
Sep 10 17:39:55.026650 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.026614    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"]
Sep 10 17:39:55.030210 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.030176    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-437dd7-wx7ll"]
Sep 10 17:39:55.060872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.060851    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n28n4\" (UniqueName: \"kubernetes.io/projected/5ed3f6bb-2d1e-406e-990f-5a674180372f-kube-api-access-n28n4\") pod \"5ed3f6bb-2d1e-406e-990f-5a674180372f\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") "
Sep 10 17:39:55.060953 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.060897    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3f6bb-2d1e-406e-990f-5a674180372f-plugins-config-volume\") pod \"5ed3f6bb-2d1e-406e-990f-5a674180372f\" (UID: \"5ed3f6bb-2d1e-406e-990f-5a674180372f\") "
Sep 10 17:39:55.061255 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.061234    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed3f6bb-2d1e-406e-990f-5a674180372f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ed3f6bb-2d1e-406e-990f-5a674180372f" (UID: "5ed3f6bb-2d1e-406e-990f-5a674180372f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:39:55.062883 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.062855    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed3f6bb-2d1e-406e-990f-5a674180372f-kube-api-access-n28n4" (OuterVolumeSpecName: "kube-api-access-n28n4") pod "5ed3f6bb-2d1e-406e-990f-5a674180372f" (UID: "5ed3f6bb-2d1e-406e-990f-5a674180372f"). InnerVolumeSpecName "kube-api-access-n28n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:39:55.161794 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.161763    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28n4\" (UniqueName: \"kubernetes.io/projected/5ed3f6bb-2d1e-406e-990f-5a674180372f-kube-api-access-n28n4\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:55.161794 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.161792    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ed3f6bb-2d1e-406e-990f-5a674180372f-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:39:55.272408 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.272324    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a2f9ee8-f70f-42b5-adf9-51a7083f4140" path="/var/lib/kubelet/pods/3a2f9ee8-f70f-42b5-adf9-51a7083f4140/volumes"
Sep 10 17:39:55.329643 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.329611    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"]
Sep 10 17:39:55.332977 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:55.332953    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-437dd7-7ffc845b7-44dxs"]
Sep 10 17:39:57.272635 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:39:57.272601    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ed3f6bb-2d1e-406e-990f-5a674180372f" path="/var/lib/kubelet/pods/5ed3f6bb-2d1e-406e-990f-5a674180372f/volumes"
Sep 10 17:40:00.403036 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:00.402985    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"]
Sep 10 17:40:15.422825 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.422778    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m" podUID="d17a5530-ebd6-4583-b15a-591d2ce38bbc" containerName="main" containerID="cri-o://2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf" gracePeriod=45
Sep 10 17:40:15.671485 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.671461    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:40:15.750426 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750397    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-home\") pod \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") "
Sep 10 17:40:15.750590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750444    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8b8f\" (UniqueName: \"kubernetes.io/projected/d17a5530-ebd6-4583-b15a-591d2ce38bbc-kube-api-access-n8b8f\") pod \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") "
Sep 10 17:40:15.750590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750465    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-model-cache\") pod \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") "
Sep 10 17:40:15.750590 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750486    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-dshm\") pod \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") "
Sep 10 17:40:15.750771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750595    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tls-certs\") pod \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") "
Sep 10 17:40:15.750771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750629    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tmp-dir\") pod \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\" (UID: \"d17a5530-ebd6-4583-b15a-591d2ce38bbc\") "
Sep 10 17:40:15.750771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750699    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-home" (OuterVolumeSpecName: "home") pod "d17a5530-ebd6-4583-b15a-591d2ce38bbc" (UID: "d17a5530-ebd6-4583-b15a-591d2ce38bbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:15.750939 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750781    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-model-cache" (OuterVolumeSpecName: "model-cache") pod "d17a5530-ebd6-4583-b15a-591d2ce38bbc" (UID: "d17a5530-ebd6-4583-b15a-591d2ce38bbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:15.751001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750933    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d17a5530-ebd6-4583-b15a-591d2ce38bbc" (UID: "d17a5530-ebd6-4583-b15a-591d2ce38bbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:15.751001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750943    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:15.751001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.750987    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:15.752603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.752577    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-dshm" (OuterVolumeSpecName: "dshm") pod "d17a5530-ebd6-4583-b15a-591d2ce38bbc" (UID: "d17a5530-ebd6-4583-b15a-591d2ce38bbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:40:15.752722 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.752629    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d17a5530-ebd6-4583-b15a-591d2ce38bbc" (UID: "d17a5530-ebd6-4583-b15a-591d2ce38bbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:40:15.752722 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.752683    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17a5530-ebd6-4583-b15a-591d2ce38bbc-kube-api-access-n8b8f" (OuterVolumeSpecName: "kube-api-access-n8b8f") pod "d17a5530-ebd6-4583-b15a-591d2ce38bbc" (UID: "d17a5530-ebd6-4583-b15a-591d2ce38bbc"). InnerVolumeSpecName "kube-api-access-n8b8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:15.852347 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.852309    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:15.852347 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.852337    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:15.852347 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.852348    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8b8f\" (UniqueName: \"kubernetes.io/projected/d17a5530-ebd6-4583-b15a-591d2ce38bbc-kube-api-access-n8b8f\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:15.852347 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:15.852357    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d17a5530-ebd6-4583-b15a-591d2ce38bbc-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:16.114520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.114438    2556 generic.go:356] "Generic (PLEG): container finished" podID="d17a5530-ebd6-4583-b15a-591d2ce38bbc" containerID="2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf" exitCode=0
Sep 10 17:40:16.114520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.114500    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"
Sep 10 17:40:16.114520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.114515    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m" event={"ID":"d17a5530-ebd6-4583-b15a-591d2ce38bbc","Type":"ContainerDied","Data":"2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf"}
Sep 10 17:40:16.114758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.114548    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m" event={"ID":"d17a5530-ebd6-4583-b15a-591d2ce38bbc","Type":"ContainerDied","Data":"5661f304776c732d96edb14cf444983779cbd69b1ff8e2318dc60f09f0fb25ca"}
Sep 10 17:40:16.114758 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.114565    2556 scope.go:117] "RemoveContainer" containerID="2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf"
Sep 10 17:40:16.123565 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.123547    2556 scope.go:117] "RemoveContainer" containerID="2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf"
Sep 10 17:40:16.123839 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:40:16.123822    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf\": container with ID starting with 2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf not found: ID does not exist" containerID="2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf"
Sep 10 17:40:16.123886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.123848    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf"} err="failed to get container status \"2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf\": rpc error: code = NotFound desc = could not find container \"2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf\": container with ID starting with 2ff53cbc4878ba35bd469a1a8db214af6b6694ce9c23caa6b1eed0356f8212cf not found: ID does not exist"
Sep 10 17:40:16.136312 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.136290    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"]
Sep 10 17:40:16.139848 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:16.139828    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-437dd7/ipp-route-437dd7-kserve-6d45684cff-tq69m"]
Sep 10 17:40:17.271756 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:17.271656    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17a5530-ebd6-4583-b15a-591d2ce38bbc" path="/var/lib/kubelet/pods/d17a5530-ebd6-4583-b15a-591d2ce38bbc/volumes"
Sep 10 17:40:43.794956 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.794918    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"]
Sep 10 17:40:43.795378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795307    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17a5530-ebd6-4583-b15a-591d2ce38bbc" containerName="main"
Sep 10 17:40:43.795378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795318    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17a5530-ebd6-4583-b15a-591d2ce38bbc" containerName="main"
Sep 10 17:40:43.795378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795326    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a2f9ee8-f70f-42b5-adf9-51a7083f4140" containerName="revoke-keys"
Sep 10 17:40:43.795378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795331    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a2f9ee8-f70f-42b5-adf9-51a7083f4140" containerName="revoke-keys"
Sep 10 17:40:43.795378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795357    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed3f6bb-2d1e-406e-990f-5a674180372f" containerName="payload-processing"
Sep 10 17:40:43.795378 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795362    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed3f6bb-2d1e-406e-990f-5a674180372f" containerName="payload-processing"
Sep 10 17:40:43.795606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795422    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed3f6bb-2d1e-406e-990f-5a674180372f" containerName="payload-processing"
Sep 10 17:40:43.795606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795431    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a2f9ee8-f70f-42b5-adf9-51a7083f4140" containerName="revoke-keys"
Sep 10 17:40:43.795606 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.795438    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17a5530-ebd6-4583-b15a-591d2ce38bbc" containerName="main"
Sep 10 17:40:43.798601 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.798584    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:43.801237 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.801216    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:40:43.801541 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.801523    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:40:43.806982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.806963    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"]
Sep 10 17:40:43.900277 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:43.900238    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12a3de7-a9c8-48a3-b6ef-14626180b151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9\" (UID: \"e12a3de7-a9c8-48a3-b6ef-14626180b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:44.001171 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:44.001130    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12a3de7-a9c8-48a3-b6ef-14626180b151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9\" (UID: \"e12a3de7-a9c8-48a3-b6ef-14626180b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:44.001816 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:44.001797    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12a3de7-a9c8-48a3-b6ef-14626180b151-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9\" (UID: \"e12a3de7-a9c8-48a3-b6ef-14626180b151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:44.110471 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:44.110387    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:44.383143 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:44.383112    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"]
Sep 10 17:40:44.385083 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:40:44.385054    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12a3de7_a9c8_48a3_b6ef_14626180b151.slice/crio-bcb0d21e20c972da2bbe1d86d61921897d4d5ef6d6689d862e855ef073b284a2 WatchSource:0}: Error finding container bcb0d21e20c972da2bbe1d86d61921897d4d5ef6d6689d862e855ef073b284a2: Status 404 returned error can't find the container with id bcb0d21e20c972da2bbe1d86d61921897d4d5ef6d6689d862e855ef073b284a2
Sep 10 17:40:45.272567 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:45.272531    2556 generic.go:356] "Generic (PLEG): container finished" podID="e12a3de7-a9c8-48a3-b6ef-14626180b151" containerID="505dd628dc46a913c5ce87af01b51c8a286b2e59525265aa45a9e3efad69c641" exitCode=0
Sep 10 17:40:45.273088 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:45.273063    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9" event={"ID":"e12a3de7-a9c8-48a3-b6ef-14626180b151","Type":"ContainerDied","Data":"505dd628dc46a913c5ce87af01b51c8a286b2e59525265aa45a9e3efad69c641"}
Sep 10 17:40:45.273136 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:45.273107    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9" event={"ID":"e12a3de7-a9c8-48a3-b6ef-14626180b151","Type":"ContainerStarted","Data":"bcb0d21e20c972da2bbe1d86d61921897d4d5ef6d6689d862e855ef073b284a2"}
Sep 10 17:40:46.418854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:46.418830    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:46.525040 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:46.524997    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12a3de7-a9c8-48a3-b6ef-14626180b151-maas-api-service-ca\") pod \"e12a3de7-a9c8-48a3-b6ef-14626180b151\" (UID: \"e12a3de7-a9c8-48a3-b6ef-14626180b151\") "
Sep 10 17:40:46.525473 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:46.525443    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12a3de7-a9c8-48a3-b6ef-14626180b151-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e12a3de7-a9c8-48a3-b6ef-14626180b151" (UID: "e12a3de7-a9c8-48a3-b6ef-14626180b151"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:46.626603 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:46.626520    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12a3de7-a9c8-48a3-b6ef-14626180b151-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:47.284557 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:47.284528    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9" event={"ID":"e12a3de7-a9c8-48a3-b6ef-14626180b151","Type":"ContainerDied","Data":"bcb0d21e20c972da2bbe1d86d61921897d4d5ef6d6689d862e855ef073b284a2"}
Sep 10 17:40:47.284557 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:47.284550    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"
Sep 10 17:40:47.284746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:47.284571    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb0d21e20c972da2bbe1d86d61921897d4d5ef6d6689d862e855ef073b284a2"
Sep 10 17:40:54.369457 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.369423    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"]
Sep 10 17:40:54.370004 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.369635    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw" podUID="69efec71-60e0-428b-9bc1-faf6e4d2cb0c" containerName="payload-processing" containerID="cri-o://46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151" gracePeriod=30
Sep 10 17:40:54.379076 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.379045    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"]
Sep 10 17:40:54.379253 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.379232    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg" podUID="e5eb0890-2b61-408d-b7b0-4900546b8e88" containerName="payload-pre-processing" containerID="cri-o://975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970" gracePeriod=30
Sep 10 17:40:54.551060 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.550991    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"]
Sep 10 17:40:54.554292 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.554263    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-777ae3-b84f9"]
Sep 10 17:40:54.627181 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.627127    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:40:54.648224 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.648198    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:40:54.695264 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695239    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb0890-2b61-408d-b7b0-4900546b8e88-plugins-config-volume\") pod \"e5eb0890-2b61-408d-b7b0-4900546b8e88\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") "
Sep 10 17:40:54.695388 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695276    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk2dr\" (UniqueName: \"kubernetes.io/projected/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-kube-api-access-vk2dr\") pod \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") "
Sep 10 17:40:54.695433 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695393    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfjgv\" (UniqueName: \"kubernetes.io/projected/e5eb0890-2b61-408d-b7b0-4900546b8e88-kube-api-access-sfjgv\") pod \"e5eb0890-2b61-408d-b7b0-4900546b8e88\" (UID: \"e5eb0890-2b61-408d-b7b0-4900546b8e88\") "
Sep 10 17:40:54.695476 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695453    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-plugins-config-volume\") pod \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\" (UID: \"69efec71-60e0-428b-9bc1-faf6e4d2cb0c\") "
Sep 10 17:40:54.695678 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695638    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5eb0890-2b61-408d-b7b0-4900546b8e88-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5eb0890-2b61-408d-b7b0-4900546b8e88" (UID: "e5eb0890-2b61-408d-b7b0-4900546b8e88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:54.695787 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695769    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5eb0890-2b61-408d-b7b0-4900546b8e88-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:54.695897 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.695871    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69efec71-60e0-428b-9bc1-faf6e4d2cb0c" (UID: "69efec71-60e0-428b-9bc1-faf6e4d2cb0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:40:54.697456 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.697435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-kube-api-access-vk2dr" (OuterVolumeSpecName: "kube-api-access-vk2dr") pod "69efec71-60e0-428b-9bc1-faf6e4d2cb0c" (UID: "69efec71-60e0-428b-9bc1-faf6e4d2cb0c"). InnerVolumeSpecName "kube-api-access-vk2dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:54.697518 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.697466    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5eb0890-2b61-408d-b7b0-4900546b8e88-kube-api-access-sfjgv" (OuterVolumeSpecName: "kube-api-access-sfjgv") pod "e5eb0890-2b61-408d-b7b0-4900546b8e88" (UID: "e5eb0890-2b61-408d-b7b0-4900546b8e88"). InnerVolumeSpecName "kube-api-access-sfjgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:40:54.797222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.797185    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjgv\" (UniqueName: \"kubernetes.io/projected/e5eb0890-2b61-408d-b7b0-4900546b8e88-kube-api-access-sfjgv\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:54.797222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.797219    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:54.797222 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:54.797229    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2dr\" (UniqueName: \"kubernetes.io/projected/69efec71-60e0-428b-9bc1-faf6e4d2cb0c-kube-api-access-vk2dr\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:40:55.271411 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.271378    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12a3de7-a9c8-48a3-b6ef-14626180b151" path="/var/lib/kubelet/pods/e12a3de7-a9c8-48a3-b6ef-14626180b151/volumes"
Sep 10 17:40:55.327294 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.327267    2556 generic.go:356] "Generic (PLEG): container finished" podID="69efec71-60e0-428b-9bc1-faf6e4d2cb0c" containerID="46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151" exitCode=0
Sep 10 17:40:55.327429 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.327320    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"
Sep 10 17:40:55.327429 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.327355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw" event={"ID":"69efec71-60e0-428b-9bc1-faf6e4d2cb0c","Type":"ContainerDied","Data":"46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151"}
Sep 10 17:40:55.327429 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.327410    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw" event={"ID":"69efec71-60e0-428b-9bc1-faf6e4d2cb0c","Type":"ContainerDied","Data":"46734d179f2cf5652a691fbffbb9b44f7a1aa7b9c4cb32044aaad369ca2417d9"}
Sep 10 17:40:55.327587 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.327439    2556 scope.go:117] "RemoveContainer" containerID="46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151"
Sep 10 17:40:55.328889 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.328865    2556 generic.go:356] "Generic (PLEG): container finished" podID="e5eb0890-2b61-408d-b7b0-4900546b8e88" containerID="975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970" exitCode=0
Sep 10 17:40:55.329010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.328931    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"
Sep 10 17:40:55.329010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.328951    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg" event={"ID":"e5eb0890-2b61-408d-b7b0-4900546b8e88","Type":"ContainerDied","Data":"975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970"}
Sep 10 17:40:55.329010 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.328992    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg" event={"ID":"e5eb0890-2b61-408d-b7b0-4900546b8e88","Type":"ContainerDied","Data":"0a2d05eea6c326e4efb10cc6be8b420ef5ab085cbd49ae6123134e2f22479812"}
Sep 10 17:40:55.336570 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.336556    2556 scope.go:117] "RemoveContainer" containerID="46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151"
Sep 10 17:40:55.336862 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:40:55.336838    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151\": container with ID starting with 46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151 not found: ID does not exist" containerID="46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151"
Sep 10 17:40:55.336911 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.336869    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151"} err="failed to get container status \"46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151\": rpc error: code = NotFound desc = could not find container \"46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151\": container with ID starting with 46ee6c2c90ef9ce4c3831b430459ce3d144b7f9c7456ece3194fefa7a79e9151 not found: ID does not exist"
Sep 10 17:40:55.336911 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.336885    2556 scope.go:117] "RemoveContainer" containerID="975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970"
Sep 10 17:40:55.344771 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.344751    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"]
Sep 10 17:40:55.345165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.345153    2556 scope.go:117] "RemoveContainer" containerID="975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970"
Sep 10 17:40:55.345422 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:40:55.345404    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970\": container with ID starting with 975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970 not found: ID does not exist" containerID="975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970"
Sep 10 17:40:55.345492 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.345429    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970"} err="failed to get container status \"975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970\": rpc error: code = NotFound desc = could not find container \"975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970\": container with ID starting with 975defebd90ede8f005794afeadc342da2832f62b7f5591c6b7ee48505a92970 not found: ID does not exist"
Sep 10 17:40:55.350128 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.350108    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-777ae3-5d478f8859-dfsxw"]
Sep 10 17:40:55.365058 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.365030    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"]
Sep 10 17:40:55.366702 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:55.366680    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-777ae3-85b74dd54b-v9fgg"]
Sep 10 17:40:57.273077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:57.273039    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69efec71-60e0-428b-9bc1-faf6e4d2cb0c" path="/var/lib/kubelet/pods/69efec71-60e0-428b-9bc1-faf6e4d2cb0c/volumes"
Sep 10 17:40:57.273546 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:40:57.273404    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5eb0890-2b61-408d-b7b0-4900546b8e88" path="/var/lib/kubelet/pods/e5eb0890-2b61-408d-b7b0-4900546b8e88/volumes"
Sep 10 17:41:01.591322 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.591281    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"]
Sep 10 17:41:01.591952 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.591934    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69efec71-60e0-428b-9bc1-faf6e4d2cb0c" containerName="payload-processing"
Sep 10 17:41:01.591999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.591954    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="69efec71-60e0-428b-9bc1-faf6e4d2cb0c" containerName="payload-processing"
Sep 10 17:41:01.591999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.591968    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5eb0890-2b61-408d-b7b0-4900546b8e88" containerName="payload-pre-processing"
Sep 10 17:41:01.591999 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.591977    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5eb0890-2b61-408d-b7b0-4900546b8e88" containerName="payload-pre-processing"
Sep 10 17:41:01.592102 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.592003    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12a3de7-a9c8-48a3-b6ef-14626180b151" containerName="revoke-keys"
Sep 10 17:41:01.592102 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.592011    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12a3de7-a9c8-48a3-b6ef-14626180b151" containerName="revoke-keys"
Sep 10 17:41:01.592102 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.592089    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="69efec71-60e0-428b-9bc1-faf6e4d2cb0c" containerName="payload-processing"
Sep 10 17:41:01.592195 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.592105    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12a3de7-a9c8-48a3-b6ef-14626180b151" containerName="revoke-keys"
Sep 10 17:41:01.592195 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.592116    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5eb0890-2b61-408d-b7b0-4900546b8e88" containerName="payload-pre-processing"
Sep 10 17:41:01.595844 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.595826    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:01.598971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.598944    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:41:01.598971 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.598944    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:41:01.600891 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.600867    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"]
Sep 10 17:41:01.660406 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.660363    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ff41c6-9045-450d-b3a6-9f735a2e432f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p\" (UID: \"74ff41c6-9045-450d-b3a6-9f735a2e432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:01.761782 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.761732    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ff41c6-9045-450d-b3a6-9f735a2e432f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p\" (UID: \"74ff41c6-9045-450d-b3a6-9f735a2e432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:01.762357 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.762337    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ff41c6-9045-450d-b3a6-9f735a2e432f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p\" (UID: \"74ff41c6-9045-450d-b3a6-9f735a2e432f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:01.908948 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:01.908862    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:02.178575 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:02.178550    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"]
Sep 10 17:41:02.180429 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:41:02.180399    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ff41c6_9045_450d_b3a6_9f735a2e432f.slice/crio-aa02002f74c1400b89244275e4098dd9ecaf191c916351db90ad07ed79159032 WatchSource:0}: Error finding container aa02002f74c1400b89244275e4098dd9ecaf191c916351db90ad07ed79159032: Status 404 returned error can't find the container with id aa02002f74c1400b89244275e4098dd9ecaf191c916351db90ad07ed79159032
Sep 10 17:41:02.370171 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:02.370142    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p" event={"ID":"74ff41c6-9045-450d-b3a6-9f735a2e432f","Type":"ContainerStarted","Data":"aa02002f74c1400b89244275e4098dd9ecaf191c916351db90ad07ed79159032"}
Sep 10 17:41:03.376318 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:03.376287    2556 generic.go:356] "Generic (PLEG): container finished" podID="74ff41c6-9045-450d-b3a6-9f735a2e432f" containerID="d9d94749f0662a3cd29cd84e809b0ee1cf38e0f9aa4b6df504bbde57415759be" exitCode=0
Sep 10 17:41:03.376697 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:03.376376    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p" event={"ID":"74ff41c6-9045-450d-b3a6-9f735a2e432f","Type":"ContainerDied","Data":"d9d94749f0662a3cd29cd84e809b0ee1cf38e0f9aa4b6df504bbde57415759be"}
Sep 10 17:41:04.509179 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:04.509158    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:04.585994 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:04.585968    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ff41c6-9045-450d-b3a6-9f735a2e432f-maas-api-service-ca\") pod \"74ff41c6-9045-450d-b3a6-9f735a2e432f\" (UID: \"74ff41c6-9045-450d-b3a6-9f735a2e432f\") "
Sep 10 17:41:04.586290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:04.586263    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ff41c6-9045-450d-b3a6-9f735a2e432f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "74ff41c6-9045-450d-b3a6-9f735a2e432f" (UID: "74ff41c6-9045-450d-b3a6-9f735a2e432f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:04.687109 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:04.687049    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74ff41c6-9045-450d-b3a6-9f735a2e432f-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:41:05.389574 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:05.389540    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"
Sep 10 17:41:05.389574 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:05.389550    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p" event={"ID":"74ff41c6-9045-450d-b3a6-9f735a2e432f","Type":"ContainerDied","Data":"aa02002f74c1400b89244275e4098dd9ecaf191c916351db90ad07ed79159032"}
Sep 10 17:41:05.389819 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:05.389600    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa02002f74c1400b89244275e4098dd9ecaf191c916351db90ad07ed79159032"
Sep 10 17:41:12.194017 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.193981    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"]
Sep 10 17:41:12.196239 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.196213    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-be583c-29817690-lmjtq"]
Sep 10 17:41:12.227177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.227143    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"]
Sep 10 17:41:12.227453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.227382    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws" podUID="32f6e8fc-1237-43ef-a536-b3dd7aa5b362" containerName="payload-processing" containerID="cri-o://5e1741c5cae1824d8b19c4f2691eb7dcf56d90efd8dcb6ca65b93df7815e342e" gracePeriod=30
Sep 10 17:41:12.235853 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.235719    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"]
Sep 10 17:41:12.236072 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.236048    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4" podUID="cadaf1a8-8e25-4536-b652-a61a0a16afb3" containerName="payload-pre-processing" containerID="cri-o://0623d4d7c1a39e56a1e727b9fda337b07d709243483b019b8bb33dbf52d0d216" gracePeriod=30
Sep 10 17:41:12.426537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.426508    2556 generic.go:356] "Generic (PLEG): container finished" podID="32f6e8fc-1237-43ef-a536-b3dd7aa5b362" containerID="5e1741c5cae1824d8b19c4f2691eb7dcf56d90efd8dcb6ca65b93df7815e342e" exitCode=0
Sep 10 17:41:12.426696 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.426578    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws" event={"ID":"32f6e8fc-1237-43ef-a536-b3dd7aa5b362","Type":"ContainerDied","Data":"5e1741c5cae1824d8b19c4f2691eb7dcf56d90efd8dcb6ca65b93df7815e342e"}
Sep 10 17:41:12.428168 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.428145    2556 generic.go:356] "Generic (PLEG): container finished" podID="cadaf1a8-8e25-4536-b652-a61a0a16afb3" containerID="0623d4d7c1a39e56a1e727b9fda337b07d709243483b019b8bb33dbf52d0d216" exitCode=0
Sep 10 17:41:12.428251 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.428208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4" event={"ID":"cadaf1a8-8e25-4536-b652-a61a0a16afb3","Type":"ContainerDied","Data":"0623d4d7c1a39e56a1e727b9fda337b07d709243483b019b8bb33dbf52d0d216"}
Sep 10 17:41:12.464274 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.464175    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"]
Sep 10 17:41:12.467059 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.467034    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-be583c-5v49p"]
Sep 10 17:41:12.506184 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.506162    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:41:12.509508 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.509490    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:41:12.662094 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.662062    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5c6p\" (UniqueName: \"kubernetes.io/projected/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-kube-api-access-x5c6p\") pod \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") "
Sep 10 17:41:12.662290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.662116    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4txr\" (UniqueName: \"kubernetes.io/projected/cadaf1a8-8e25-4536-b652-a61a0a16afb3-kube-api-access-x4txr\") pod \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") "
Sep 10 17:41:12.662290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.662221    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-plugins-config-volume\") pod \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\" (UID: \"32f6e8fc-1237-43ef-a536-b3dd7aa5b362\") "
Sep 10 17:41:12.662290 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.662245    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadaf1a8-8e25-4536-b652-a61a0a16afb3-plugins-config-volume\") pod \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\" (UID: \"cadaf1a8-8e25-4536-b652-a61a0a16afb3\") "
Sep 10 17:41:12.662658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.662627    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32f6e8fc-1237-43ef-a536-b3dd7aa5b362" (UID: "32f6e8fc-1237-43ef-a536-b3dd7aa5b362"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:12.662658 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.662648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadaf1a8-8e25-4536-b652-a61a0a16afb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cadaf1a8-8e25-4536-b652-a61a0a16afb3" (UID: "cadaf1a8-8e25-4536-b652-a61a0a16afb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:12.664176 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.664146    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadaf1a8-8e25-4536-b652-a61a0a16afb3-kube-api-access-x4txr" (OuterVolumeSpecName: "kube-api-access-x4txr") pod "cadaf1a8-8e25-4536-b652-a61a0a16afb3" (UID: "cadaf1a8-8e25-4536-b652-a61a0a16afb3"). InnerVolumeSpecName "kube-api-access-x4txr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:12.664293 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.664272    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-kube-api-access-x5c6p" (OuterVolumeSpecName: "kube-api-access-x5c6p") pod "32f6e8fc-1237-43ef-a536-b3dd7aa5b362" (UID: "32f6e8fc-1237-43ef-a536-b3dd7aa5b362"). InnerVolumeSpecName "kube-api-access-x5c6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:41:12.763868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.763840    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:41:12.763868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.763862    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cadaf1a8-8e25-4536-b652-a61a0a16afb3-plugins-config-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:41:12.763868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.763871    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5c6p\" (UniqueName: \"kubernetes.io/projected/32f6e8fc-1237-43ef-a536-b3dd7aa5b362-kube-api-access-x5c6p\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:41:12.764063 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:12.763879    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4txr\" (UniqueName: \"kubernetes.io/projected/cadaf1a8-8e25-4536-b652-a61a0a16afb3-kube-api-access-x4txr\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:41:13.272059 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.272027    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3db1792c-8a30-4ed8-9ee8-806ed1238596" path="/var/lib/kubelet/pods/3db1792c-8a30-4ed8-9ee8-806ed1238596/volumes"
Sep 10 17:41:13.272435 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.272329    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ff41c6-9045-450d-b3a6-9f735a2e432f" path="/var/lib/kubelet/pods/74ff41c6-9045-450d-b3a6-9f735a2e432f/volumes"
Sep 10 17:41:13.434686 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.434634    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"
Sep 10 17:41:13.434868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.434692    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4" event={"ID":"cadaf1a8-8e25-4536-b652-a61a0a16afb3","Type":"ContainerDied","Data":"643678f943d743486f893a521d3024ffddab4d3abf3023315413b173e01f94ed"}
Sep 10 17:41:13.434868 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.434747    2556 scope.go:117] "RemoveContainer" containerID="0623d4d7c1a39e56a1e727b9fda337b07d709243483b019b8bb33dbf52d0d216"
Sep 10 17:41:13.436137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.436115    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"
Sep 10 17:41:13.436241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.436121    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws" event={"ID":"32f6e8fc-1237-43ef-a536-b3dd7aa5b362","Type":"ContainerDied","Data":"88fa5f2aac28766107f3a80c3ce50cea20fa34804afb826779090932ed7fb23f"}
Sep 10 17:41:13.446568 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.446546    2556 scope.go:117] "RemoveContainer" containerID="5e1741c5cae1824d8b19c4f2691eb7dcf56d90efd8dcb6ca65b93df7815e342e"
Sep 10 17:41:13.457854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.457833    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"]
Sep 10 17:41:13.463166 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.463146    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-be583c-58b98f9b9d-l8tm4"]
Sep 10 17:41:13.474009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.473988    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"]
Sep 10 17:41:13.478285 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:13.478261    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-be583c-6d4848d999-qxrws"]
Sep 10 17:41:15.272289 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:15.272253    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f6e8fc-1237-43ef-a536-b3dd7aa5b362" path="/var/lib/kubelet/pods/32f6e8fc-1237-43ef-a536-b3dd7aa5b362/volumes"
Sep 10 17:41:15.272681 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:15.272594    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cadaf1a8-8e25-4536-b652-a61a0a16afb3" path="/var/lib/kubelet/pods/cadaf1a8-8e25-4536-b652-a61a0a16afb3/volumes"
Sep 10 17:41:19.413362 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413328    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"]
Sep 10 17:41:19.413793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413714    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ff41c6-9045-450d-b3a6-9f735a2e432f" containerName="revoke-keys"
Sep 10 17:41:19.413793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413726    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ff41c6-9045-450d-b3a6-9f735a2e432f" containerName="revoke-keys"
Sep 10 17:41:19.413793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413743    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f6e8fc-1237-43ef-a536-b3dd7aa5b362" containerName="payload-processing"
Sep 10 17:41:19.413793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413748    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f6e8fc-1237-43ef-a536-b3dd7aa5b362" containerName="payload-processing"
Sep 10 17:41:19.413793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413760    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cadaf1a8-8e25-4536-b652-a61a0a16afb3" containerName="payload-pre-processing"
Sep 10 17:41:19.413793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413766    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cadaf1a8-8e25-4536-b652-a61a0a16afb3" containerName="payload-pre-processing"
Sep 10 17:41:19.414020 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413836    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ff41c6-9045-450d-b3a6-9f735a2e432f" containerName="revoke-keys"
Sep 10 17:41:19.414020 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413845    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f6e8fc-1237-43ef-a536-b3dd7aa5b362" containerName="payload-processing"
Sep 10 17:41:19.414020 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.413852    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cadaf1a8-8e25-4536-b652-a61a0a16afb3" containerName="payload-pre-processing"
Sep 10 17:41:19.418522 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.418502    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:19.421359 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.421339    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:41:19.421499 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.421341    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:41:19.425007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.424979    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dace7ce-9463-4992-83a1-f5c5302e4cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9\" (UID: \"5dace7ce-9463-4992-83a1-f5c5302e4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:19.425120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.425077    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"]
Sep 10 17:41:19.525600 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.525565    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dace7ce-9463-4992-83a1-f5c5302e4cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9\" (UID: \"5dace7ce-9463-4992-83a1-f5c5302e4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:19.526224 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.526206    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dace7ce-9463-4992-83a1-f5c5302e4cb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9\" (UID: \"5dace7ce-9463-4992-83a1-f5c5302e4cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:19.730685 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:19.730628    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:20.000943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:20.000921    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"]
Sep 10 17:41:20.003027 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:41:20.002998    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dace7ce_9463_4992_83a1_f5c5302e4cb2.slice/crio-0251c6db4f8477e434df2a94888d8320ace5a2dfe5cccc13258d81965fb10fb8 WatchSource:0}: Error finding container 0251c6db4f8477e434df2a94888d8320ace5a2dfe5cccc13258d81965fb10fb8: Status 404 returned error can't find the container with id 0251c6db4f8477e434df2a94888d8320ace5a2dfe5cccc13258d81965fb10fb8
Sep 10 17:41:20.477165 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:20.477115    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9" event={"ID":"5dace7ce-9463-4992-83a1-f5c5302e4cb2","Type":"ContainerStarted","Data":"f71a1cc1a97bfa25479af9a0487e8fbaaab8a5fd4b06a02da173010803ce5455"}
Sep 10 17:41:20.477609 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:20.477185    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9" event={"ID":"5dace7ce-9463-4992-83a1-f5c5302e4cb2","Type":"ContainerStarted","Data":"0251c6db4f8477e434df2a94888d8320ace5a2dfe5cccc13258d81965fb10fb8"}
Sep 10 17:41:20.497536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:20.497496    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9" podStartSLOduration=1.497481678 podStartE2EDuration="1.497481678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:41:20.496261567 +0000 UTC m=+1765.716385500" watchObservedRunningTime="2026-09-10 17:41:20.497481678 +0000 UTC m=+1765.717605612"
Sep 10 17:41:21.484196 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:21.484164    2556 generic.go:356] "Generic (PLEG): container finished" podID="5dace7ce-9463-4992-83a1-f5c5302e4cb2" containerID="f71a1cc1a97bfa25479af9a0487e8fbaaab8a5fd4b06a02da173010803ce5455" exitCode=0
Sep 10 17:41:21.484591 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:21.484216    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9" event={"ID":"5dace7ce-9463-4992-83a1-f5c5302e4cb2","Type":"ContainerDied","Data":"f71a1cc1a97bfa25479af9a0487e8fbaaab8a5fd4b06a02da173010803ce5455"}
Sep 10 17:41:22.618907 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:22.618883    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:22.652686 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:22.652644    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dace7ce-9463-4992-83a1-f5c5302e4cb2-maas-api-service-ca\") pod \"5dace7ce-9463-4992-83a1-f5c5302e4cb2\" (UID: \"5dace7ce-9463-4992-83a1-f5c5302e4cb2\") "
Sep 10 17:41:22.652953 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:22.652933    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dace7ce-9463-4992-83a1-f5c5302e4cb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dace7ce-9463-4992-83a1-f5c5302e4cb2" (UID: "5dace7ce-9463-4992-83a1-f5c5302e4cb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 17:41:22.753747 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:22.753691    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dace7ce-9463-4992-83a1-f5c5302e4cb2-maas-api-service-ca\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:41:23.496392 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:23.496365    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"
Sep 10 17:41:23.496584 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:23.496364    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9" event={"ID":"5dace7ce-9463-4992-83a1-f5c5302e4cb2","Type":"ContainerDied","Data":"0251c6db4f8477e434df2a94888d8320ace5a2dfe5cccc13258d81965fb10fb8"}
Sep 10 17:41:23.496584 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:23.496489    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0251c6db4f8477e434df2a94888d8320ace5a2dfe5cccc13258d81965fb10fb8"
Sep 10 17:41:30.283323 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:30.283289    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"]
Sep 10 17:41:30.287116 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:30.287091    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-6806f1-54cl9"]
Sep 10 17:41:31.271293 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:31.271260    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dace7ce-9463-4992-83a1-f5c5302e4cb2" path="/var/lib/kubelet/pods/5dace7ce-9463-4992-83a1-f5c5302e4cb2/volumes"
Sep 10 17:41:55.556399 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:55.556367    2556 scope.go:117] "RemoveContainer" containerID="7aebdaa40658d179d21fb28e8b6743183690e48ab6972d0a8611cdd9c3653c27"
Sep 10 17:41:55.564873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:55.564855    2556 scope.go:117] "RemoveContainer" containerID="6f48c92e2cd22c141138ad0815c91fcecc487d05c6bb1ad060b14a3135527351"
Sep 10 17:41:55.572524 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:41:55.572513    2556 scope.go:117] "RemoveContainer" containerID="f2a2c84f39bcfcb2518a7aa48271974e5c9c5f727ac4f00603e171eb05a3b99a"
Sep 10 17:42:55.606183 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:42:55.606149    2556 scope.go:117] "RemoveContainer" containerID="c6e2b732d7aeff527fed57b7840978a83bbee9828aeb0fc2661e62e7fed95c75"
Sep 10 17:43:55.628433 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:43:55.628397    2556 scope.go:117] "RemoveContainer" containerID="003f0c9998b98e45b9912868421c4f5ff1402a56bee785f06559698f54144f7c"
Sep 10 17:44:22.544625 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.544582    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"]
Sep 10 17:44:22.545251 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.544875    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr" podUID="a4831a7a-ae3b-42d1-9484-8752ed63bd1b" containerName="manager" containerID="cri-o://13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b" gracePeriod=10
Sep 10 17:44:22.801416 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.801344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:44:22.890645 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.890615    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7j85\" (UniqueName: \"kubernetes.io/projected/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-kube-api-access-z7j85\") pod \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") "
Sep 10 17:44:22.890833 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.890681    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-extensions-socket-volume\") pod \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\" (UID: \"a4831a7a-ae3b-42d1-9484-8752ed63bd1b\") "
Sep 10 17:44:22.891077 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.891042    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a4831a7a-ae3b-42d1-9484-8752ed63bd1b" (UID: "a4831a7a-ae3b-42d1-9484-8752ed63bd1b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:44:22.892719 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.892697    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-kube-api-access-z7j85" (OuterVolumeSpecName: "kube-api-access-z7j85") pod "a4831a7a-ae3b-42d1-9484-8752ed63bd1b" (UID: "a4831a7a-ae3b-42d1-9484-8752ed63bd1b"). InnerVolumeSpecName "kube-api-access-z7j85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:44:22.992007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.991964    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7j85\" (UniqueName: \"kubernetes.io/projected/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-kube-api-access-z7j85\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:44:22.992007 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:22.991997    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a4831a7a-ae3b-42d1-9484-8752ed63bd1b-extensions-socket-volume\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:44:23.411772 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.411738    2556 generic.go:356] "Generic (PLEG): container finished" podID="a4831a7a-ae3b-42d1-9484-8752ed63bd1b" containerID="13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b" exitCode=0
Sep 10 17:44:23.411949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.411782    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"
Sep 10 17:44:23.411949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.411808    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr" event={"ID":"a4831a7a-ae3b-42d1-9484-8752ed63bd1b","Type":"ContainerDied","Data":"13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b"}
Sep 10 17:44:23.411949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.411850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr" event={"ID":"a4831a7a-ae3b-42d1-9484-8752ed63bd1b","Type":"ContainerDied","Data":"b4d99597bd209a94bbdf7f1973786cf8d0e0a9d91932fd5b364703884a56068c"}
Sep 10 17:44:23.411949 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.411872    2556 scope.go:117] "RemoveContainer" containerID="13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b"
Sep 10 17:44:23.421739 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.421712    2556 scope.go:117] "RemoveContainer" containerID="13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b"
Sep 10 17:44:23.422014 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:44:23.421993    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b\": container with ID starting with 13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b not found: ID does not exist" containerID="13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b"
Sep 10 17:44:23.422063 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.422025    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b"} err="failed to get container status \"13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b\": rpc error: code = NotFound desc = could not find container \"13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b\": container with ID starting with 13a73888ad5ea167589f7250bf88d2d334fa0e0c946086bd91b283f43cbb686b not found: ID does not exist"
Sep 10 17:44:23.431297 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.431270    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"]
Sep 10 17:44:23.435898 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:23.435878    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-sntdr"]
Sep 10 17:44:25.271724 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:44:25.271689    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4831a7a-ae3b-42d1-9484-8752ed63bd1b" path="/var/lib/kubelet/pods/a4831a7a-ae3b-42d1-9484-8752ed63bd1b/volumes"
Sep 10 17:45:00.156043 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156003    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"]
Sep 10 17:45:00.156491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156398    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dace7ce-9463-4992-83a1-f5c5302e4cb2" containerName="revoke-keys"
Sep 10 17:45:00.156491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156411    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dace7ce-9463-4992-83a1-f5c5302e4cb2" containerName="revoke-keys"
Sep 10 17:45:00.156491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156426    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4831a7a-ae3b-42d1-9484-8752ed63bd1b" containerName="manager"
Sep 10 17:45:00.156491 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156431    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4831a7a-ae3b-42d1-9484-8752ed63bd1b" containerName="manager"
Sep 10 17:45:00.156653 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156496    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dace7ce-9463-4992-83a1-f5c5302e4cb2" containerName="revoke-keys"
Sep 10 17:45:00.156653 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.156508    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4831a7a-ae3b-42d1-9484-8752ed63bd1b" containerName="manager"
Sep 10 17:45:00.159435 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.159411    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"
Sep 10 17:45:00.165106 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.165083    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4mln5\""
Sep 10 17:45:00.169224 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.169202    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"
Sep 10 17:45:00.172418 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.172395    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"]
Sep 10 17:45:00.444426 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.444391    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"]
Sep 10 17:45:00.445987 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:45:00.445959    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18977315_16ab_4a54_9ad8_4b315399916c.slice/crio-ffa2088c41b622293560c8cd60194eeb541093f4cc1c493b9810bb43a263719e WatchSource:0}: Error finding container ffa2088c41b622293560c8cd60194eeb541093f4cc1c493b9810bb43a263719e: Status 404 returned error can't find the container with id ffa2088c41b622293560c8cd60194eeb541093f4cc1c493b9810bb43a263719e
Sep 10 17:45:00.592379 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:00.592342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx" event={"ID":"18977315-16ab-4a54-9ad8-4b315399916c","Type":"ContainerStarted","Data":"ffa2088c41b622293560c8cd60194eeb541093f4cc1c493b9810bb43a263719e"}
Sep 10 17:45:01.599278 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:01.599186    2556 generic.go:356] "Generic (PLEG): container finished" podID="18977315-16ab-4a54-9ad8-4b315399916c" containerID="cf80d996d75a0eec0fe6c9db9d8be44ccf8403ef2ec4e1c9c39042a4406fc7fe" exitCode=0
Sep 10 17:45:01.599640 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:01.599266    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx" event={"ID":"18977315-16ab-4a54-9ad8-4b315399916c","Type":"ContainerDied","Data":"cf80d996d75a0eec0fe6c9db9d8be44ccf8403ef2ec4e1c9c39042a4406fc7fe"}
Sep 10 17:45:02.730311 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:02.730287    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"
Sep 10 17:45:03.610086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:03.610001    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx" event={"ID":"18977315-16ab-4a54-9ad8-4b315399916c","Type":"ContainerDied","Data":"ffa2088c41b622293560c8cd60194eeb541093f4cc1c493b9810bb43a263719e"}
Sep 10 17:45:03.610086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:03.610051    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa2088c41b622293560c8cd60194eeb541093f4cc1c493b9810bb43a263719e"
Sep 10 17:45:03.610086 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:03.610016    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817705-rtnfx"
Sep 10 17:45:23.241940 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.241900    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"]
Sep 10 17:45:23.242324 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.242313    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18977315-16ab-4a54-9ad8-4b315399916c" containerName="cleanup"
Sep 10 17:45:23.242370 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.242326    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="18977315-16ab-4a54-9ad8-4b315399916c" containerName="cleanup"
Sep 10 17:45:23.242409 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.242398    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="18977315-16ab-4a54-9ad8-4b315399916c" containerName="cleanup"
Sep 10 17:45:23.245573 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.245553    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.248130 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.248110    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jfn2p\""
Sep 10 17:45:23.259053 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.259030    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"]
Sep 10 17:45:23.362608 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.362571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppmmj\" (UniqueName: \"kubernetes.io/projected/3de566b3-4fe3-4d60-9484-b3b86a363634-kube-api-access-ppmmj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zfldl\" (UID: \"3de566b3-4fe3-4d60-9484-b3b86a363634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.362812 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.362634    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de566b3-4fe3-4d60-9484-b3b86a363634-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zfldl\" (UID: \"3de566b3-4fe3-4d60-9484-b3b86a363634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.463874 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.463836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppmmj\" (UniqueName: \"kubernetes.io/projected/3de566b3-4fe3-4d60-9484-b3b86a363634-kube-api-access-ppmmj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zfldl\" (UID: \"3de566b3-4fe3-4d60-9484-b3b86a363634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.464044 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.463901    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de566b3-4fe3-4d60-9484-b3b86a363634-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zfldl\" (UID: \"3de566b3-4fe3-4d60-9484-b3b86a363634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.464286 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.464265    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3de566b3-4fe3-4d60-9484-b3b86a363634-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zfldl\" (UID: \"3de566b3-4fe3-4d60-9484-b3b86a363634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.473035 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.473010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppmmj\" (UniqueName: \"kubernetes.io/projected/3de566b3-4fe3-4d60-9484-b3b86a363634-kube-api-access-ppmmj\") pod \"kuadrant-operator-controller-manager-5769f99d9f-zfldl\" (UID: \"3de566b3-4fe3-4d60-9484-b3b86a363634\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.555442 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.555369    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:23.829285 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:23.829244    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"]
Sep 10 17:45:23.831112 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:45:23.831088    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de566b3_4fe3_4d60_9484_b3b86a363634.slice/crio-12f142d94349ecd29caa3848560312b3286f749cb69263767243edfe6fcf460e WatchSource:0}: Error finding container 12f142d94349ecd29caa3848560312b3286f749cb69263767243edfe6fcf460e: Status 404 returned error can't find the container with id 12f142d94349ecd29caa3848560312b3286f749cb69263767243edfe6fcf460e
Sep 10 17:45:24.713340 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:24.713304    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl" event={"ID":"3de566b3-4fe3-4d60-9484-b3b86a363634","Type":"ContainerStarted","Data":"dd1c4c396214f4ceb026d8187b967b409ed480205e8d614b762b4e398c40e02c"}
Sep 10 17:45:24.713773 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:24.713362    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl" event={"ID":"3de566b3-4fe3-4d60-9484-b3b86a363634","Type":"ContainerStarted","Data":"12f142d94349ecd29caa3848560312b3286f749cb69263767243edfe6fcf460e"}
Sep 10 17:45:24.713773 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:24.713386    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:24.737130 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:24.737068    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl" podStartSLOduration=1.737052484 podStartE2EDuration="1.737052484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:45:24.736453514 +0000 UTC m=+2009.956577456" watchObservedRunningTime="2026-09-10 17:45:24.737052484 +0000 UTC m=+2009.957176416"
Sep 10 17:45:35.720865 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:35.720832    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-zfldl"
Sep 10 17:45:55.659453 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:55.659418    2556 scope.go:117] "RemoveContainer" containerID="d4992f305dd94534e29639d1a0c0d4b1a606e7f81f2a46588cd94b56489ee1dc"
Sep 10 17:45:55.667710 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:45:55.667691    2556 scope.go:117] "RemoveContainer" containerID="33a61b35bcfd0106ff0ea8cdfd01253dff1d5b68ef241358e1cf7e101c775434"
Sep 10 17:46:11.545784 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.545749    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"]
Sep 10 17:46:11.549963 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.549943    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.552507 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.552484    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0c44a93d-kserve-self-signed-certs\""
Sep 10 17:46:11.558401 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.558379    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"]
Sep 10 17:46:11.606268 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.606237    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tls-certs\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.606393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.606302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkqw6\" (UniqueName: \"kubernetes.io/projected/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-kube-api-access-tkqw6\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.606393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.606338    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-home\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.606393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.606367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tmp-dir\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.606393 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.606386    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-dshm\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.606536 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.606416    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-model-cache\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.709104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.708312    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-model-cache\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.709104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.708406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tls-certs\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.709104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.708496    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkqw6\" (UniqueName: \"kubernetes.io/projected/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-kube-api-access-tkqw6\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.709104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.708538    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-home\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.709104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.708593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tmp-dir\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.709104 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.708630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-dshm\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.712597 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.711987    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-dshm\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.712597 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.712253    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-model-cache\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.714182 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.714159    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-home\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.715169 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.715130    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tmp-dir\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.715380 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.715362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tls-certs\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.724921 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.724894    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkqw6\" (UniqueName: \"kubernetes.io/projected/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-kube-api-access-tkqw6\") pod \"e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") " pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:11.860137 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:11.860024    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:12.131094 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:12.130968    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"]
Sep 10 17:46:12.133400 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:46:12.133374    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f5b6e2b_f732_40dd_87ff_90ab4564bb55.slice/crio-39109f6cb20878c046c987857f2e61c129093c75cd91f141b595ff4da6adc42a WatchSource:0}: Error finding container 39109f6cb20878c046c987857f2e61c129093c75cd91f141b595ff4da6adc42a: Status 404 returned error can't find the container with id 39109f6cb20878c046c987857f2e61c129093c75cd91f141b595ff4da6adc42a
Sep 10 17:46:12.955873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:12.955830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5" event={"ID":"9f5b6e2b-f732-40dd-87ff-90ab4564bb55","Type":"ContainerStarted","Data":"0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad"}
Sep 10 17:46:12.956269 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:12.955883    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5" event={"ID":"9f5b6e2b-f732-40dd-87ff-90ab4564bb55","Type":"ContainerStarted","Data":"39109f6cb20878c046c987857f2e61c129093c75cd91f141b595ff4da6adc42a"}
Sep 10 17:46:12.974088 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:12.974038    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5" podStartSLOduration=1.974024235 podStartE2EDuration="1.974024235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:12.97204789 +0000 UTC m=+2058.192171826" watchObservedRunningTime="2026-09-10 17:46:12.974024235 +0000 UTC m=+2058.194148168"
Sep 10 17:46:21.861198 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:21.861114    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:21.861198 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:21.861174    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:21.873154 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:21.873124    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:22.011998 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:22.011965    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:46:25.067997 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.067956    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"]
Sep 10 17:46:25.071982 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.071963    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.075091 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.075070    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0c44a93d-kserve-self-signed-certs\""
Sep 10 17:46:25.086326 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.086302    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"]
Sep 10 17:46:25.244177 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.244137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tls-certs\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.244348 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.244210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tmp-dir\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.244348 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.244284    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmr67\" (UniqueName: \"kubernetes.io/projected/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-kube-api-access-bmr67\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.244348 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.244330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-model-cache\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.244463 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.244427    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-home\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.244463 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.244456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-dshm\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345415 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345324    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tls-certs\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345415 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345375    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tmp-dir\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345423    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmr67\" (UniqueName: \"kubernetes.io/projected/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-kube-api-access-bmr67\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345469    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-model-cache\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345543    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-home\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345567    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-dshm\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345931 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345908    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tmp-dir\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.345981 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345957    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-model-cache\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.346033 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.345980    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-home\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.347785 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.347761    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-dshm\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.347969 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.347950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tls-certs\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.359002 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.358979    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmr67\" (UniqueName: \"kubernetes.io/projected/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-kube-api-access-bmr67\") pod \"e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") " pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.381142 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.381121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:25.653396 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:25.653371    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"]
Sep 10 17:46:25.655284 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:46:25.655256    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03d5f97e_1ce4_4078_8be0_cf5d0b87ff1b.slice/crio-ffd25c09dec685c0b7cf37438cbb55a6d2db175241c251e3402a21826d920ab3 WatchSource:0}: Error finding container ffd25c09dec685c0b7cf37438cbb55a6d2db175241c251e3402a21826d920ab3: Status 404 returned error can't find the container with id ffd25c09dec685c0b7cf37438cbb55a6d2db175241c251e3402a21826d920ab3
Sep 10 17:46:26.023069 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:26.023030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd" event={"ID":"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b","Type":"ContainerStarted","Data":"ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd"}
Sep 10 17:46:26.023069 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:26.023084    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd" event={"ID":"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b","Type":"ContainerStarted","Data":"ffd25c09dec685c0b7cf37438cbb55a6d2db175241c251e3402a21826d920ab3"}
Sep 10 17:46:26.051690 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:26.051624    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd" podStartSLOduration=1.0515975850000001 podStartE2EDuration="1.051597585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:46:26.049209486 +0000 UTC m=+2071.269333439" watchObservedRunningTime="2026-09-10 17:46:26.051597585 +0000 UTC m=+2071.271721517"
Sep 10 17:46:35.381607 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:35.381564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:35.382021 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:35.381627    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:35.394306 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:35.394278    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:36.087597 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:36.087566    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:46:52.572494 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:52.572460    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"]
Sep 10 17:46:54.564294 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:54.564260    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"]
Sep 10 17:46:55.692131 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:46:55.692099    2556 scope.go:117] "RemoveContainer" containerID="505dd628dc46a913c5ce87af01b51c8a286b2e59525265aa45a9e3efad69c641"
Sep 10 17:47:07.591843 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.591797    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5" podUID="9f5b6e2b-f732-40dd-87ff-90ab4564bb55" containerName="main" containerID="cri-o://0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad" gracePeriod=45
Sep 10 17:47:07.840704 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.840675    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:47:07.855407 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855325    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tmp-dir\") pod \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") "
Sep 10 17:47:07.855407 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855392    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-dshm\") pod \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") "
Sep 10 17:47:07.855596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855453    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-model-cache\") pod \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") "
Sep 10 17:47:07.855596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855496    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-home\") pod \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") "
Sep 10 17:47:07.855596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855549    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tls-certs\") pod \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") "
Sep 10 17:47:07.855596 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855573    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkqw6\" (UniqueName: \"kubernetes.io/projected/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-kube-api-access-tkqw6\") pod \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\" (UID: \"9f5b6e2b-f732-40dd-87ff-90ab4564bb55\") "
Sep 10 17:47:07.855827 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855585    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f5b6e2b-f732-40dd-87ff-90ab4564bb55" (UID: "9f5b6e2b-f732-40dd-87ff-90ab4564bb55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:07.855827 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855785    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f5b6e2b-f732-40dd-87ff-90ab4564bb55" (UID: "9f5b6e2b-f732-40dd-87ff-90ab4564bb55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:07.855827 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855801    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-home" (OuterVolumeSpecName: "home") pod "9f5b6e2b-f732-40dd-87ff-90ab4564bb55" (UID: "9f5b6e2b-f732-40dd-87ff-90ab4564bb55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:07.855986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855937    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:07.855986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855956    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:07.855986 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.855971    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:07.857886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.857854    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f5b6e2b-f732-40dd-87ff-90ab4564bb55" (UID: "9f5b6e2b-f732-40dd-87ff-90ab4564bb55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:47:07.858158 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.858124    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-dshm" (OuterVolumeSpecName: "dshm") pod "9f5b6e2b-f732-40dd-87ff-90ab4564bb55" (UID: "9f5b6e2b-f732-40dd-87ff-90ab4564bb55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:07.858158 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.858135    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-kube-api-access-tkqw6" (OuterVolumeSpecName: "kube-api-access-tkqw6") pod "9f5b6e2b-f732-40dd-87ff-90ab4564bb55" (UID: "9f5b6e2b-f732-40dd-87ff-90ab4564bb55"). InnerVolumeSpecName "kube-api-access-tkqw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:47:07.956462 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.956420    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:07.956462 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.956455    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkqw6\" (UniqueName: \"kubernetes.io/projected/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-kube-api-access-tkqw6\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:07.956462 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:07.956468    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f5b6e2b-f732-40dd-87ff-90ab4564bb55-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:08.236047 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.236016    2556 generic.go:356] "Generic (PLEG): container finished" podID="9f5b6e2b-f732-40dd-87ff-90ab4564bb55" containerID="0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad" exitCode=0
Sep 10 17:47:08.236219 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.236056    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"
Sep 10 17:47:08.236219 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.236102    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5" event={"ID":"9f5b6e2b-f732-40dd-87ff-90ab4564bb55","Type":"ContainerDied","Data":"0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad"}
Sep 10 17:47:08.236219 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.236145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5" event={"ID":"9f5b6e2b-f732-40dd-87ff-90ab4564bb55","Type":"ContainerDied","Data":"39109f6cb20878c046c987857f2e61c129093c75cd91f141b595ff4da6adc42a"}
Sep 10 17:47:08.236219 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.236164    2556 scope.go:117] "RemoveContainer" containerID="0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad"
Sep 10 17:47:08.246018 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.246002    2556 scope.go:117] "RemoveContainer" containerID="0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad"
Sep 10 17:47:08.246288 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:47:08.246271    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad\": container with ID starting with 0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad not found: ID does not exist" containerID="0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad"
Sep 10 17:47:08.246363 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.246293    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad"} err="failed to get container status \"0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad\": rpc error: code = NotFound desc = could not find container \"0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad\": container with ID starting with 0b62d7669b46dc368c07ccec1fdb4c672bb1387d0fb42da48f18a1de4a9519ad not found: ID does not exist"
Sep 10 17:47:08.265209 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.265188    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"]
Sep 10 17:47:08.282410 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:08.282389    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0c44a93d-kserve-56d76b5cdc-gqsv5"]
Sep 10 17:47:09.271231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.271195    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f5b6e2b-f732-40dd-87ff-90ab4564bb55" path="/var/lib/kubelet/pods/9f5b6e2b-f732-40dd-87ff-90ab4564bb55/volumes"
Sep 10 17:47:09.583720 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.583603    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd" podUID="03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" containerName="main" containerID="cri-o://ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd" gracePeriod=45
Sep 10 17:47:09.834890 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.834829    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:47:09.870062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870023    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-dshm\") pod \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") "
Sep 10 17:47:09.870231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870083    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tmp-dir\") pod \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") "
Sep 10 17:47:09.870231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870106    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-model-cache\") pod \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") "
Sep 10 17:47:09.870231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870154    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmr67\" (UniqueName: \"kubernetes.io/projected/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-kube-api-access-bmr67\") pod \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") "
Sep 10 17:47:09.870231 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870199    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-home\") pod \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") "
Sep 10 17:47:09.870430 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870242    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tls-certs\") pod \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\" (UID: \"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b\") "
Sep 10 17:47:09.870430 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870363    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" (UID: "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:09.870539 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870428    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" (UID: "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:09.870539 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870523    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tmp-dir\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:09.870648 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870546    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-model-cache\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:09.870648 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.870520    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-home" (OuterVolumeSpecName: "home") pod "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" (UID: "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:09.872241 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.872215    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-dshm" (OuterVolumeSpecName: "dshm") pod "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" (UID: "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 17:47:09.872356 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.872320    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-kube-api-access-bmr67" (OuterVolumeSpecName: "kube-api-access-bmr67") pod "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" (UID: "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b"). InnerVolumeSpecName "kube-api-access-bmr67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 17:47:09.872356 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.872345    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" (UID: "03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 17:47:09.972009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.971971    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmr67\" (UniqueName: \"kubernetes.io/projected/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-kube-api-access-bmr67\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:09.972009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.972004    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-home\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:09.972009 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.972014    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-tls-certs\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:09.972252 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:09.972022    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b-dshm\") on node \"ip-10-0-137-19.ec2.internal\" DevicePath \"\""
Sep 10 17:47:10.249930 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.249892    2556 generic.go:356] "Generic (PLEG): container finished" podID="03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" containerID="ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd" exitCode=0
Sep 10 17:47:10.250106 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.249943    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"
Sep 10 17:47:10.250106 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.249979    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd" event={"ID":"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b","Type":"ContainerDied","Data":"ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd"}
Sep 10 17:47:10.250106 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.250031    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd" event={"ID":"03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b","Type":"ContainerDied","Data":"ffd25c09dec685c0b7cf37438cbb55a6d2db175241c251e3402a21826d920ab3"}
Sep 10 17:47:10.250106 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.250052    2556 scope.go:117] "RemoveContainer" containerID="ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd"
Sep 10 17:47:10.261140 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.261117    2556 scope.go:117] "RemoveContainer" containerID="ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd"
Sep 10 17:47:10.261612 ip-10-0-137-19 kubenswrapper[2556]: E0910 17:47:10.261591    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd\": container with ID starting with ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd not found: ID does not exist" containerID="ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd"
Sep 10 17:47:10.261719 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.261619    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd"} err="failed to get container status \"ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd\": rpc error: code = NotFound desc = could not find container \"ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd\": container with ID starting with ee00d8bfc7f98a5a56912ad17d611941dea94a6b3bf4fb1ac91a1c256785aacd not found: ID does not exist"
Sep 10 17:47:10.284312 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.284285    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"]
Sep 10 17:47:10.296730 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:10.296705    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0c44a93d-kserve-769cfb9b67-grgrd"]
Sep 10 17:47:11.272854 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:11.272818    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" path="/var/lib/kubelet/pods/03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b/volumes"
Sep 10 17:47:18.495115 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495075    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"]
Sep 10 17:47:18.495768 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495739    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f5b6e2b-f732-40dd-87ff-90ab4564bb55" containerName="main"
Sep 10 17:47:18.495768 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495765    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f5b6e2b-f732-40dd-87ff-90ab4564bb55" containerName="main"
Sep 10 17:47:18.495943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495806    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" containerName="main"
Sep 10 17:47:18.495943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495816    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" containerName="main"
Sep 10 17:47:18.495943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495900    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f5b6e2b-f732-40dd-87ff-90ab4564bb55" containerName="main"
Sep 10 17:47:18.495943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.495908    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d5f97e-1ce4-4078-8be0-cf5d0b87ff1b" containerName="main"
Sep 10 17:47:18.500202 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.500185    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.504284 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.504262    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 17:47:18.504533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.504513    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 17:47:18.504533 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.504527    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 17:47:18.504708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.504542    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-n7zfj\""
Sep 10 17:47:18.504708 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.504592    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 17:47:18.516120 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.516091    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"]
Sep 10 17:47:18.547450 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.547420    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/321ed465-80a5-4999-9b63-41c95b8de9aa-kube-api-access-tqwh6\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.547571 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.547483    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321ed465-80a5-4999-9b63-41c95b8de9aa-maas-api-tls\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.547616 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.547561    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321ed465-80a5-4999-9b63-41c95b8de9aa-metrics-certs\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.547654 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.547614    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321ed465-80a5-4999-9b63-41c95b8de9aa-openshift-service-ca\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.648562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.648523    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/321ed465-80a5-4999-9b63-41c95b8de9aa-kube-api-access-tqwh6\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.648749 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.648611    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321ed465-80a5-4999-9b63-41c95b8de9aa-maas-api-tls\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.648749 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.648653    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321ed465-80a5-4999-9b63-41c95b8de9aa-metrics-certs\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.648749 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.648707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321ed465-80a5-4999-9b63-41c95b8de9aa-openshift-service-ca\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.649425 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.649403    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/321ed465-80a5-4999-9b63-41c95b8de9aa-openshift-service-ca\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.651197 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.651167    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/321ed465-80a5-4999-9b63-41c95b8de9aa-maas-api-tls\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.651308 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.651286    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/321ed465-80a5-4999-9b63-41c95b8de9aa-metrics-certs\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.657570 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.657550    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwh6\" (UniqueName: \"kubernetes.io/projected/321ed465-80a5-4999-9b63-41c95b8de9aa-kube-api-access-tqwh6\") pod \"maas-api-777cfbf6dc-kd6tz\" (UID: \"321ed465-80a5-4999-9b63-41c95b8de9aa\") " pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:18.810763 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:18.810651    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:19.085488 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:19.085462    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"]
Sep 10 17:47:19.087310 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:47:19.087274    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321ed465_80a5_4999_9b63_41c95b8de9aa.slice/crio-834549a12f741f82c8eeb9edadfed3d6364ec0673ebecdfb9e26b1d39138b430 WatchSource:0}: Error finding container 834549a12f741f82c8eeb9edadfed3d6364ec0673ebecdfb9e26b1d39138b430: Status 404 returned error can't find the container with id 834549a12f741f82c8eeb9edadfed3d6364ec0673ebecdfb9e26b1d39138b430
Sep 10 17:47:19.088678 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:19.088642    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 17:47:19.293944 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:19.293900    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz" event={"ID":"321ed465-80a5-4999-9b63-41c95b8de9aa","Type":"ContainerStarted","Data":"834549a12f741f82c8eeb9edadfed3d6364ec0673ebecdfb9e26b1d39138b430"}
Sep 10 17:47:22.311707 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:22.311638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz" event={"ID":"321ed465-80a5-4999-9b63-41c95b8de9aa","Type":"ContainerStarted","Data":"d8b706311484b2612feaff613275e3507505803eb74477cc500f3d812534387e"}
Sep 10 17:47:22.312129 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:22.311736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:22.332228 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:22.332178    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz" podStartSLOduration=2.13842111 podStartE2EDuration="4.332160721s" totalImagesPullingTime="2.193739611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 17:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:47:22.330217087 +0000 UTC m=+2127.550341034" watchObservedRunningTime="2026-09-10 17:47:22.332160721 +0000 UTC m=+2127.552284654"
Sep 10 17:47:28.322246 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:28.322211    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-777cfbf6dc-kd6tz"
Sep 10 17:47:41.474082 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:41.474044    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584cbdc55b-c779r_aeedba45-d8c6-44df-9182-e25748511025/authorino/0.log"
Sep 10 17:47:47.357183 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:47.357115    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-ftdv7_884349a5-f6b9-47be-bc67-41aa70754038/copy-manifests/0.log"
Sep 10 17:47:47.371001 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:47.370975    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-ftdv7_884349a5-f6b9-47be-bc67-41aa70754038/manager/0.log"
Sep 10 17:47:48.085933 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:48.085901    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-777cfbf6dc-kd6tz_321ed465-80a5-4999-9b63-41c95b8de9aa/maas-api/0.log"
Sep 10 17:47:48.307886 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:48.307858    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817705-rtnfx_18977315-16ab-4a54-9ad8-4b315399916c/cleanup/0.log"
Sep 10 17:47:49.199408 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:49.199375    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584cbdc55b-c779r_aeedba45-d8c6-44df-9182-e25748511025/authorino/0.log"
Sep 10 17:47:49.652037 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:49.652004    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zfldl_3de566b3-4fe3-4d60-9484-b3b86a363634/manager/0.log"
Sep 10 17:47:50.451966 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:50.451931    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647997b56d-8xz8b_06f4db00-e12f-4410-8ec1-4d334ad686a9/kube-auth-proxy/0.log"
Sep 10 17:47:50.807169 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:50.807132    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4kjb5_391069a5-647f-4d8d-8111-7d6caa56d2a2/payload-pre-processing/0.log"
Sep 10 17:47:50.920517 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:50.920487    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wjvg7_97929e09-8ffb-4a86-b440-b2bed7e9a3ac/payload-processing/0.log"
Sep 10 17:47:51.603608 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:51.603580    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-nx7kq_5bed72cb-5e79-4c62-ae6f-2ecfca10c4eb/main/0.log"
Sep 10 17:47:52.077246 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:52.077215    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-g7jqk_ed0715c7-1ba9-4a17-b5be-e164f774f20e/main/0.log"
Sep 10 17:47:55.711776 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:55.711652    2556 scope.go:117] "RemoveContainer" containerID="f71a1cc1a97bfa25479af9a0487e8fbaaab8a5fd4b06a02da173010803ce5455"
Sep 10 17:47:55.720872 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:47:55.720851    2556 scope.go:117] "RemoveContainer" containerID="d9d94749f0662a3cd29cd84e809b0ee1cf38e0f9aa4b6df504bbde57415759be"
Sep 10 17:48:00.318408 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:00.318376    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6fwnl_29c79ac8-60f9-434a-9687-28a323a15418/global-pull-secret-syncer/0.log"
Sep 10 17:48:00.489356 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:00.489324    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nnk4v_46a93d23-62be-48a7-ad9c-ffa56649cbcd/konnectivity-agent/0.log"
Sep 10 17:48:00.512374 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:00.512338    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-19.ec2.internal_70b70169d86c3bcf2d0a1e114ed46d9a/haproxy/0.log"
Sep 10 17:48:05.194867 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:05.194827    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584cbdc55b-c779r_aeedba45-d8c6-44df-9182-e25748511025/authorino/0.log"
Sep 10 17:48:05.354376 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:05.354338    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-zfldl_3de566b3-4fe3-4d60-9484-b3b86a363634/manager/0.log"
Sep 10 17:48:06.921804 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:06.921775    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/alertmanager/0.log"
Sep 10 17:48:06.965821 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:06.965789    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/config-reloader/0.log"
Sep 10 17:48:06.993093 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:06.993065    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/kube-rbac-proxy-web/0.log"
Sep 10 17:48:07.018366 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.018338    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/kube-rbac-proxy/0.log"
Sep 10 17:48:07.043527 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.043497    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/kube-rbac-proxy-metric/0.log"
Sep 10 17:48:07.069383 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.069352    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/prom-label-proxy/0.log"
Sep 10 17:48:07.095384 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.095305    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_417d8e16-cdbd-4d5d-a693-83d4988718a0/init-config-reloader/0.log"
Sep 10 17:48:07.296615 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.296584    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-96dmx_0d04454e-e78a-4736-86fd-9cf2cc563221/monitoring-plugin/0.log"
Sep 10 17:48:07.530497 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.530473    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gpztw_599f330f-e7b2-4cc2-998a-8a3de3084b88/node-exporter/0.log"
Sep 10 17:48:07.558964 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.558933    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gpztw_599f330f-e7b2-4cc2-998a-8a3de3084b88/kube-rbac-proxy/0.log"
Sep 10 17:48:07.591537 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.591511    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gpztw_599f330f-e7b2-4cc2-998a-8a3de3084b88/init-textfile/0.log"
Sep 10 17:48:07.622288 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.622261    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vk9ms_0352af2d-c1da-4b0a-bff1-5f54daaa0a17/kube-rbac-proxy-main/0.log"
Sep 10 17:48:07.656206 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.656176    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vk9ms_0352af2d-c1da-4b0a-bff1-5f54daaa0a17/kube-rbac-proxy-self/0.log"
Sep 10 17:48:07.682479 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.682449    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-vk9ms_0352af2d-c1da-4b0a-bff1-5f54daaa0a17/openshift-state-metrics/0.log"
Sep 10 17:48:07.753247 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.753214    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/prometheus/0.log"
Sep 10 17:48:07.791422 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.791344    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/config-reloader/0.log"
Sep 10 17:48:07.820926 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.820901    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/thanos-sidecar/0.log"
Sep 10 17:48:07.856049 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.856021    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/kube-rbac-proxy-web/0.log"
Sep 10 17:48:07.887444 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.887417    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/kube-rbac-proxy/0.log"
Sep 10 17:48:07.913057 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.913003    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/kube-rbac-proxy-thanos/0.log"
Sep 10 17:48:07.940639 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:07.940614    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_40433bd3-fc5e-4d50-ba4e-f4f2a628e8e6/init-config-reloader/0.log"
Sep 10 17:48:08.038013 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.037984    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-8hsfl_0fb6ec77-0a92-4946-a40c-5a35b6a14649/prometheus-operator-admission-webhook/0.log"
Sep 10 17:48:08.077279 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.077169    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d6c7bd9d-s8bjd_837c5b96-d480-4059-ac9f-88ec5262ebd7/telemeter-client/0.log"
Sep 10 17:48:08.109895 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.109837    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d6c7bd9d-s8bjd_837c5b96-d480-4059-ac9f-88ec5262ebd7/reload/0.log"
Sep 10 17:48:08.135783 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.135726    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78d6c7bd9d-s8bjd_837c5b96-d480-4059-ac9f-88ec5262ebd7/kube-rbac-proxy/0.log"
Sep 10 17:48:08.177482 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.177448    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b49f6574f-kxbmr_d6db4dea-e1d1-425c-9486-50297d78e145/thanos-query/0.log"
Sep 10 17:48:08.214985 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.214925    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b49f6574f-kxbmr_d6db4dea-e1d1-425c-9486-50297d78e145/kube-rbac-proxy-web/0.log"
Sep 10 17:48:08.240386 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.240356    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b49f6574f-kxbmr_d6db4dea-e1d1-425c-9486-50297d78e145/kube-rbac-proxy/0.log"
Sep 10 17:48:08.266543 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.266514    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b49f6574f-kxbmr_d6db4dea-e1d1-425c-9486-50297d78e145/prom-label-proxy/0.log"
Sep 10 17:48:08.293144 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.293112    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b49f6574f-kxbmr_d6db4dea-e1d1-425c-9486-50297d78e145/kube-rbac-proxy-rules/0.log"
Sep 10 17:48:08.321915 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:08.321883    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b49f6574f-kxbmr_d6db4dea-e1d1-425c-9486-50297d78e145/kube-rbac-proxy-metrics/0.log"
Sep 10 17:48:09.724873 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:09.724828    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-jtw2d_5bf26107-29a9-488c-8d83-f60f2cc0d540/networking-console-plugin/0.log"
Sep 10 17:48:10.488910 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.488873    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"]
Sep 10 17:48:10.492889 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.492836    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.495520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.495498    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlqxs\"/\"openshift-service-ca.crt\""
Sep 10 17:48:10.496508 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.496489    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mlqxs\"/\"kube-root-ca.crt\""
Sep 10 17:48:10.496508 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.496499    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mlqxs\"/\"default-dockercfg-trhqt\""
Sep 10 17:48:10.504520 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.504497    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"]
Sep 10 17:48:10.637368 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.637333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8kj\" (UniqueName: \"kubernetes.io/projected/fdb5f0c6-c709-452b-b927-5570b3924d1d-kube-api-access-6b8kj\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.637562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.637390    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-lib-modules\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.637562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.637458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-sys\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.637562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.637527    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-podres\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.637562 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.637549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-proc\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738305 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8kj\" (UniqueName: \"kubernetes.io/projected/fdb5f0c6-c709-452b-b927-5570b3924d1d-kube-api-access-6b8kj\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-lib-modules\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-sys\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738396    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-podres\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-proc\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738490    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-sys\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738503    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-proc\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738532    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-lib-modules\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.738793 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.738560    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fdb5f0c6-c709-452b-b927-5570b3924d1d-podres\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.747426 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.747368    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8kj\" (UniqueName: \"kubernetes.io/projected/fdb5f0c6-c709-452b-b927-5570b3924d1d-kube-api-access-6b8kj\") pod \"perf-node-gather-daemonset-rg82p\" (UID: \"fdb5f0c6-c709-452b-b927-5570b3924d1d\") " pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:10.802746 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:10.802708    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:11.073418 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:11.073389    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"]
Sep 10 17:48:11.075117 ip-10-0-137-19 kubenswrapper[2556]: W0910 17:48:11.075086    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfdb5f0c6_c709_452b_b927_5570b3924d1d.slice/crio-dedb5e5a04426d68a4feab6eab366f1e27e9a8c8693328e76d655be9f3ef299d WatchSource:0}: Error finding container dedb5e5a04426d68a4feab6eab366f1e27e9a8c8693328e76d655be9f3ef299d: Status 404 returned error can't find the container with id dedb5e5a04426d68a4feab6eab366f1e27e9a8c8693328e76d655be9f3ef299d
Sep 10 17:48:11.565062 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:11.565025    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p" event={"ID":"fdb5f0c6-c709-452b-b927-5570b3924d1d","Type":"ContainerStarted","Data":"85ebaf647ddee0e0877fa9ac8444aea70fab0a2accb65b00f322dc81475372f6"}
Sep 10 17:48:11.565249 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:11.565085    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p" event={"ID":"fdb5f0c6-c709-452b-b927-5570b3924d1d","Type":"ContainerStarted","Data":"dedb5e5a04426d68a4feab6eab366f1e27e9a8c8693328e76d655be9f3ef299d"}
Sep 10 17:48:11.565249 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:11.565114    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:11.583232 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:11.583182    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p" podStartSLOduration=1.583168881 podStartE2EDuration="1.583168881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 17:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 17:48:11.581876836 +0000 UTC m=+2176.802000769" watchObservedRunningTime="2026-09-10 17:48:11.583168881 +0000 UTC m=+2176.803292813"
Sep 10 17:48:12.905147 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:12.905115    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rh8lm_10b2638b-880e-4a9a-b835-687ccc17feee/dns/0.log"
Sep 10 17:48:12.930087 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:12.930057    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rh8lm_10b2638b-880e-4a9a-b835-687ccc17feee/kube-rbac-proxy/0.log"
Sep 10 17:48:13.073065 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:13.073035    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nxqp5_14bbd0d2-4568-41dc-ad8f-6129d10314f8/dns-node-resolver/0.log"
Sep 10 17:48:13.619175 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:13.619143    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5987474868-bvs76_3f7f7c1e-58f8-4634-bcbc-fdf08e55e41e/registry/0.log"
Sep 10 17:48:13.646559 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:13.646534    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2t5g6_9be1264a-5add-475d-9fe4-adcf3db28e24/node-ca/0.log"
Sep 10 17:48:14.990678 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:14.990647    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647997b56d-8xz8b_06f4db00-e12f-4410-8ec1-4d334ad686a9/kube-auth-proxy/0.log"
Sep 10 17:48:15.106567 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:15.106534    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4kjb5_391069a5-647f-4d8d-8111-7d6caa56d2a2/payload-pre-processing/0.log"
Sep 10 17:48:15.149542 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:15.149510    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wjvg7_97929e09-8ffb-4a86-b440-b2bed7e9a3ac/payload-processing/0.log"
Sep 10 17:48:16.075913 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:16.075881    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7mbmg_c7766669-96fc-4947-b998-827e01940bc8/serve-healthcheck-canary/0.log"
Sep 10 17:48:16.667271 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:16.667244    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-44wdl_9d5c734b-d5a6-43ba-9da7-f298828e310c/kube-rbac-proxy/0.log"
Sep 10 17:48:16.693293 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:16.693268    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-44wdl_9d5c734b-d5a6-43ba-9da7-f298828e310c/exporter/0.log"
Sep 10 17:48:16.726256 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:16.726226    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-44wdl_9d5c734b-d5a6-43ba-9da7-f298828e310c/extractor/0.log"
Sep 10 17:48:17.580139 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:17.580114    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mlqxs/perf-node-gather-daemonset-rg82p"
Sep 10 17:48:19.829943 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:19.829910    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-ftdv7_884349a5-f6b9-47be-bc67-41aa70754038/manager/0.log"
Sep 10 17:48:19.856097 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:19.856068    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-86f7548996-ftdv7_884349a5-f6b9-47be-bc67-41aa70754038/copy-manifests/0.log"
Sep 10 17:48:21.584903 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:21.584870    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9fc458d94-phwhm_eca88365-fadb-443f-9f5a-663ddae8deae/manager/0.log"
Sep 10 17:48:21.631837 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:21.631804    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647997b56d-8xz8b_06f4db00-e12f-4410-8ec1-4d334ad686a9/kube-auth-proxy/0.log"
Sep 10 17:48:21.713232 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:21.713204    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4kjb5_391069a5-647f-4d8d-8111-7d6caa56d2a2/payload-pre-processing/0.log"
Sep 10 17:48:21.744750 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:21.744721    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wjvg7_97929e09-8ffb-4a86-b440-b2bed7e9a3ac/payload-processing/0.log"
Sep 10 17:48:23.566530 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:23.566501    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-647997b56d-8xz8b_06f4db00-e12f-4410-8ec1-4d334ad686a9/kube-auth-proxy/0.log"
Sep 10 17:48:23.657615 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:23.657588    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-4kjb5_391069a5-647f-4d8d-8111-7d6caa56d2a2/payload-pre-processing/0.log"
Sep 10 17:48:23.682802 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:23.682763    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wjvg7_97929e09-8ffb-4a86-b440-b2bed7e9a3ac/payload-processing/0.log"
Sep 10 17:48:26.990784 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:26.990754    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xzfp5_ba152442-477b-4c0f-959e-690826579bd1/migrator/0.log"
Sep 10 17:48:27.017145 ip-10-0-137-19 kubenswrapper[2556]: I0910 17:48:27.017115    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-xzfp5_ba152442-477b-4c0f-959e-690826579bd1/graceful-termination/0.log"