Sep 16 12:38:17.787729 ip-10-0-143-72 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:38:18.246915 ip-10-0-143-72 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:38:18.246915 ip-10-0-143-72 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:38:18.246915 ip-10-0-143-72 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:38:18.246915 ip-10-0-143-72 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:38:18.246915 ip-10-0-143-72 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:38:18.248134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.248021    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:38:18.252165 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252148    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252166    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252170    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252173    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252176    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252179    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252182    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252184    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252187    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252190    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252192    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252195    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252197    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252200    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252208    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:18.252205 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252211    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252215    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252217    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252220    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252222    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252226    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252228    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252231    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252233    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252236    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252238    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252241    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252243    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252247    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252252    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252255    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252257    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252260    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252262    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:18.252586 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252265    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252267    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252269    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252272    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252275    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252277    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252279    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252282    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252285    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252288    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252290    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252292    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252295    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252297    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252299    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252311    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252315    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252335    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252339    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252343    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:18.253036 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252346    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252348    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252353    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252356    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252360    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252363    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252366    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252369    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252371    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252374    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252377    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252379    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252382    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252385    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252387    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252390    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252392    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252396    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252399    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:18.253534 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252402    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252404    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252407    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252409    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252412    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252415    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252417    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252421    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252425    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252428    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252431    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252433    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252435    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252438    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252440    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252443    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252446    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252449    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252451    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252454    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:18.254033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252456    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252459    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252461    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252465    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252469    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.252472    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253145    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253152    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253155    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253158    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253161    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253164    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253167    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253169    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253172    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253174    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253177    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253180    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253183    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253186    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:18.254524 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253189    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253191    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253194    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253196    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253199    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253202    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253204    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253207    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253209    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253212    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253214    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253217    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253219    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253222    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253224    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253227    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253229    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253232    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253234    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253237    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:18.255024 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253239    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253242    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253244    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253247    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253249    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253252    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253254    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253257    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253260    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253263    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253267    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253270    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253273    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253276    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253280    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253282    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253285    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253288    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253290    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:18.255525 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253293    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253295    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253298    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253300    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253303    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253305    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253308    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253310    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253313    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253315    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253333    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253337    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253339    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253342    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253344    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253347    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253350    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253352    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253356    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253359    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:18.255991 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253362    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253364    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253367    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253371    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253376    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253380    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253383    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253386    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253388    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253391    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253393    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253398    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253401    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253404    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253407    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253409    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253413    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253415    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253418    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:18.256510 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253420    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253423    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253425    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253428    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253431    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253433    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253435    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253797    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253803    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253805    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253808    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253811    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253814    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253819    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253823    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253826    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253829    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253832    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253835    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:18.256974 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253838    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253841    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253844    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253847    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253849    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253852    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253854    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253857    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253859    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253862    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253864    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253867    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253869    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253872    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253874    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253877    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253879    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253882    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253884    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253886    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253889    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:18.257546 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253892    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253894    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253896    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253901    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253904    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253907    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253910    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253912    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253915    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253917    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253920    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253923    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253926    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253929    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253931    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253935    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253938    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253941    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253944    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:18.258061 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253947    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253949    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253953    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253956    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253958    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253961    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253964    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253966    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253969    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253971    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253974    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253976    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253979    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253982    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253984    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253986    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253990    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253992    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.253996    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254000    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:18.258538 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254002    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254005    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254007    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254011    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254013    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254016    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254018    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254021    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254024    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254026    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254029    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254031    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254034    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254036    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254039    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254041    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254044    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254047    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254049    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254052    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:18.259054 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254054    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254057    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254059    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254062    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254064    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254066    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254069    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254147    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254183    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254189    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254194    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254200    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254204    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254208    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254212    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254215    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254218    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254222    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254225    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254228    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254231    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254234    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254237    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:38:18.259532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254240    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254242    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254247    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254250    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254253    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254256    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254259    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254263    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254279    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254282    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254286    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254289    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254292    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254295    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254298    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254301    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254305    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254309    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254312    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254315    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254336    2559 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254341    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254347    2559 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254350    2559 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:38:18.260093 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254353    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254356    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254359    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254362    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254366    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254368    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254371    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254374    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254377    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254380    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254383    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254386    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254389    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254391    2559 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254395    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254398    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254401    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254404    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254407    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254410    2559 flags.go:64] FLAG: --help="false"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254412    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254415    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254418    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:38:18.260699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254421    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254424    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254429    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254432    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254434    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254439    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254442    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254445    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254448    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254451    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254454    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254457    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254460    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254463    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254465    2559 flags.go:64] FLAG: --lock-file=""
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254468    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254471    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254474    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254479    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254482    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254485    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254488    2559 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254491    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254494    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:38:18.261272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254496    2559 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254499    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254503    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254506    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254511    2559 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254514    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254516    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254520    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254522    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254526    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254530    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254533    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254540    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254544    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254547    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254550    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254553    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254562    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254565    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254568    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254571    2559 flags.go:64] FLAG: --port="10250"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254574    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254577    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03b2541561e3aedce"
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254580    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:38:18.261870 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254583    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254586    2559 flags.go:64] FLAG: --register-node="true"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254589    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254592    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254595    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254598    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254600    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254606    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254610    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254613    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254616    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254618    2559 flags.go:64] FLAG: --runonce="false"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254621    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254624    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254627    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254630    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254633    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254636    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254639    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254643    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254645    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254648    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254652    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254655    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254658    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:38:18.262511 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254664    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254667    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254672    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254675    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254678    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254682    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254685    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254688    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254691    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254693    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254696    2559 flags.go:64] FLAG: --v="2"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254700    2559 flags.go:64] FLAG: --version="false"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254704    2559 flags.go:64] FLAG: --vmodule=""
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254708    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.254711    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254844    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254848    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254852    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254855    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254858    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254861    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254864    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254866    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:18.263131 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254869    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254871    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254874    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254877    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254879    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254882    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254886    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254888    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254891    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254894    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254897    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254900    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254902    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254905    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254908    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254910    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254915    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254918    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254921    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:18.263712 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254924    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254927    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254930    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254933    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254935    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254938    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254941    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254944    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254947    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254949    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254952    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254955    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254957    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254960    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254962    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254965    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254967    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254970    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254972    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254977    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:18.264180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254979    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254982    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254985    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254987    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254990    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254993    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254996    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.254998    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255001    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255003    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255006    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255008    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255011    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255013    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255015    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255018    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255021    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255023    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255026    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255028    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:18.264784 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255031    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255034    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255036    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255039    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255043    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255046    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255049    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255051    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255054    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255056    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255059    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255063    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255066    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255068    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255071    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255074    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255076    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255080    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255082    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255085    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:18.265272 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255087    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255090    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255092    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255095    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255099    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255103    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255105    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255108    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255111    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255114    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255117    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.255120    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.256037    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.263902    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.263916    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:38:18.265769 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.264924    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.264935    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.264995    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.264999    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265004    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265007    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265010    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265013    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265015    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265020    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265023    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265026    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265029    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265031    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265035    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265038    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265041    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265044    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265047    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:18.266184 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265050    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265053    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265055    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265058    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265060    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265063    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265067    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265071    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265074    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265077    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265080    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265083    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265086    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265089    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265092    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265095    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265098    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265100    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265103    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:18.266663 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265105    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265108    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265110    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265113    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265116    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265118    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265121    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265123    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265126    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265130    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265133    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265136    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265139    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265142    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265146    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265150    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265152    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265155    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265158    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:18.267132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265160    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265163    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265165    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265168    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265170    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265172    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265175    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265177    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265180    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265182    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265185    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265187    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265190    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265192    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265195    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265197    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265200    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265203    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265205    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265207    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:18.267596 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265210    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265212    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265215    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265218    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265220    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265223    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265225    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265228    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265230    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265233    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265236    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265238    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265240    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265243    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265245    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265248    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265250    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265253    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265255    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:18.268073 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265257    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265260    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265262    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265265    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265267    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.265272    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265399    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265404    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265408    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265412    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265415    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265418    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265421    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265424    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265427    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:38:18.268620 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265430    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265433    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265436    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265439    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265441    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265444    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265447    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265450    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265454    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265456    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265459    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265461    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265464    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265466    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265469    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265471    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265473    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265476    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265478    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:38:18.268988 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265480    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265483    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265485    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265488    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265491    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265493    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265496    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265498    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265501    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265503    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265506    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265508    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265511    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265513    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265516    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265518    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265521    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265524    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265526    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:38:18.269460 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265529    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265531    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265534    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265536    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265539    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265541    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265543    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265546    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265548    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265552    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265555    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265558    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265561    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265563    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265566    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265569    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265572    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265574    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265577    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265580    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:38:18.269908 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265582    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265584    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265587    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265589    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265592    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265594    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265597    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265599    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265602    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265604    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265607    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265610    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265613    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265615    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265617    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265620    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265622    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265625    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265627    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265629    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:38:18.270397 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265632    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265634    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265637    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265639    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265641    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265644    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265646    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265648    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265651    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265654    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265656    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:18.265658    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.265663    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.265773    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.269039    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:38:18.270876 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.270132    2559 server.go:1019] "Starting client certificate rotation"
Sep 16 12:38:18.271236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.270238    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:38:18.271236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.271091    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:38:18.275108 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.275086    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:38:18.277370 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.277353    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:38:18.285392 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.285278    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:38:18.288731 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.288712    2559 log.go:25] "Validated CRI v1 image API"
Sep 16 12:38:18.290287 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.290073    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:38:18.293573 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.293551    2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d345405c-42bf-4072-ad55-6a3bec97833e:/dev/nvme0n1p4 dc57d634-a1ae-4769-89ca-895709169e04:/dev/nvme0n1p3]
Sep 16 12:38:18.293637 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.293572    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:38:18.298475 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.298366    2559 manager.go:216] Machine: {Timestamp:2026-09-16 12:38:18.296947483 +0000 UTC m=+0.380390508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098546 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273695bc5a6c08691d158a078f9e3b SystemUUID:ec273695-bc5a-6c08-691d-158a078f9e3b BootID:6e58921e-2bb6-45fb-949e-410664eb3e90 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ea:f8:58:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ea:f8:58:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:aa:19:ba:f9:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:38:18.298475 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.298469    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:38:18.298584 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.298570    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:38:18.302121 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.302099    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:38:18.302259 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.302125    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-72.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:38:18.302954 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.302943    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:38:18.302994 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.302956    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:38:18.302994 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.302963    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:38:18.303736 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.303726    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:38:18.304934 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.304921    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:38:18.309643 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309631    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:38:18.309756 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309747    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:38:18.309794 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309781    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:38:18.309824 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309806    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:38:18.309862 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309829    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:38:18.309862 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309843    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:38:18.309912 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.309866    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:38:18.310259 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.310248    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:38:18.310301 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.310268    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:38:18.312664 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.312648    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:38:18.312878 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.312862    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:38:18.316753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.316731    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:38:18.316829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.316768    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:38:18.318818 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318801    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318823    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318829    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318834    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318842    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318848    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318861    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318867    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318873    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318878    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318886    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:38:18.318895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.318891    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:38:18.319816 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.319805    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:38:18.319816 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.319815    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:38:18.323613 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.323595    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-72.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:38:18.323613 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.323606    2559 server.go:1268] "Started kubelet"
Sep 16 12:38:18.323735 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.323632    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:38:18.323735 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.323696    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:38:18.323813 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.323722    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:38:18.324021 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.323994    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:38:18.324171 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.324153    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:38:18.324382 ip-10-0-143-72 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:38:18.324885 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.324866    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:38:18.325920 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.325905    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:38:18.327961 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.327942    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:38:18.333381 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.333364    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:38:18.333464 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.333411    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:38:18.333744 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.333562    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.334361 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.334343    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:38:18.334476 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.334460    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:38:18.334580 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.334351    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:38:18.334643 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.334611    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:38:18.334643 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.334619    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:38:18.335715 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.335699    2559 factory.go:55] Registering systemd factory
Sep 16 12:38:18.335715 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.335719    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:38:18.335957 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.335934    2559 factory.go:153] Registering CRI-O factory
Sep 16 12:38:18.335957 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.335948    2559 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:38:18.336034 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.335986    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:38:18.336034 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.336000    2559 factory.go:103] Registering Raw factory
Sep 16 12:38:18.336034 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.336010    2559 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:38:18.336333 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.336312    2559 manager.go:349] Starting recovery of all containers
Sep 16 12:38:18.337060 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.337039    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-72.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 12:38:18.337400 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.337380    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 12:38:18.338425 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.337194    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-72.ec2.internal.18d5cd6dc70234ec  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-72.ec2.internal,UID:ip-10-0-143-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-72.ec2.internal,},FirstTimestamp:2026-09-16 12:38:18.32358014 +0000 UTC m=+0.407023143,LastTimestamp:2026-09-16 12:38:18.32358014 +0000 UTC m=+0.407023143,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-72.ec2.internal,}"
Sep 16 12:38:18.344936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.344824    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jdw54"
Sep 16 12:38:18.349108 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.349092    2559 manager.go:354] Recovery completed
Sep 16 12:38:18.353333 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.353307    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jdw54"
Sep 16 12:38:18.353491 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.353479    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.355295 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355279    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.355394 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355312    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.355394 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355344    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.355907 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355892    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:38:18.355907 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355907    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:38:18.356016 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355921    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:38:18.356016 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.355895    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.357432 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.357413    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.357512 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.357445    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.357512 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.357463    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.357512 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.357433    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-72.ec2.internal.18d5cd6dc8e62807  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-72.ec2.internal,UID:ip-10-0-143-72.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-72.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-72.ec2.internal,},FirstTimestamp:2026-09-16 12:38:18.355296263 +0000 UTC m=+0.438739272,LastTimestamp:2026-09-16 12:38:18.355296263 +0000 UTC m=+0.438739272,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-72.ec2.internal,}"
Sep 16 12:38:18.357733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.357722    2559 policy_none.go:49] "None policy: Start"
Sep 16 12:38:18.357777 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.357737    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:38:18.357777 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.357747    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:38:18.359673 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.359662    2559 policy_none.go:47] "Start"
Sep 16 12:38:18.397905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.397885    2559 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.397924    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.397935    2559 server.go:87] "Starting device plugin registration server"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.398254    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.398288    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.398493    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.398714    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.398726    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.399346    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.399396    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.403782    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.405149    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.405198    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.405224    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.405264    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:38:18.414134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.407280    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:18.498971 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.498844    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.499958 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.499943    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.500027 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.499973    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.500027 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.499983    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.500027 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.500005    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.506081 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.506063    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"]
Sep 16 12:38:18.506136 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.506129    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.507745 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.507729    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.507819 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.507759    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.507819 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.507768    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.508872 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.508856    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.508917 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.508891    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-72.ec2.internal\": node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.510084 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510072    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.510200 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510185    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.510236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510214    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.510815 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510799    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.510880 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510826    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.510880 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510840    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.510880 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510861    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.510880 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510877    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.511014 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.510889    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.513046 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.513031    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.513080 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.513061    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:38:18.514556 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.514538    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:38:18.514668 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.514569    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:38:18.514668 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.514580    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:38:18.529524 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.529504    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.541925 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.541899    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-72.ec2.internal\" not found" node="ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.546332 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.546294    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-72.ec2.internal\" not found" node="ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.629647 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.629620    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.634931 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.634910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be62b2323e6968ec6b10000746a16003-config\") pod \"kube-apiserver-proxy-ip-10-0-143-72.ec2.internal\" (UID: \"be62b2323e6968ec6b10000746a16003\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.635046 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.634936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fd67f18f3d247683da7ebab0e2c5954-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal\" (UID: \"1fd67f18f3d247683da7ebab0e2c5954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.635046 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.634953    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fd67f18f3d247683da7ebab0e2c5954-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal\" (UID: \"1fd67f18f3d247683da7ebab0e2c5954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.730693 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.730662    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.736022 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.736004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be62b2323e6968ec6b10000746a16003-config\") pod \"kube-apiserver-proxy-ip-10-0-143-72.ec2.internal\" (UID: \"be62b2323e6968ec6b10000746a16003\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.736074 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.736031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fd67f18f3d247683da7ebab0e2c5954-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal\" (UID: \"1fd67f18f3d247683da7ebab0e2c5954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.736074 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.736050    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fd67f18f3d247683da7ebab0e2c5954-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal\" (UID: \"1fd67f18f3d247683da7ebab0e2c5954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.736139 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.736078    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/be62b2323e6968ec6b10000746a16003-config\") pod \"kube-apiserver-proxy-ip-10-0-143-72.ec2.internal\" (UID: \"be62b2323e6968ec6b10000746a16003\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.736171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.736132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1fd67f18f3d247683da7ebab0e2c5954-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal\" (UID: \"1fd67f18f3d247683da7ebab0e2c5954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.736171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.736132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1fd67f18f3d247683da7ebab0e2c5954-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal\" (UID: \"1fd67f18f3d247683da7ebab0e2c5954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.831510 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.831440    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:18.845880 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.845862    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.850037 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:18.850018    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:18.932526 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:18.932490    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.032996 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.032966    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.133538 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.133456    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.234001 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.233965    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.270401 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.270369    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:38:19.270775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.270540    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:38:19.334058 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.334023    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.347790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.347765    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:19.357766 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.357727    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 12:33:19 +0000 UTC" deadline="2028-04-05 16:59:25.185316078 +0000 UTC"
Sep 16 12:38:19.357766 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.357760    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13612h21m5.827558968s"
Sep 16 12:38:19.434490 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.434455    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.526127 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:19.526092    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd67f18f3d247683da7ebab0e2c5954.slice/crio-58e40ac79cb23e994234d5f1fc961f912eb2e69f73931b26457f851065de7118 WatchSource:0}: Error finding container 58e40ac79cb23e994234d5f1fc961f912eb2e69f73931b26457f851065de7118: Status 404 returned error can't find the container with id 58e40ac79cb23e994234d5f1fc961f912eb2e69f73931b26457f851065de7118
Sep 16 12:38:19.530257 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.530242    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:38:19.534955 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.534938    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.557908 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.557887    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:19.571536 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:19.571515    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe62b2323e6968ec6b10000746a16003.slice/crio-c50506537a24c33c8181be8531fd13c7b9686325ab09dfa1dcb9ad484bf294cf WatchSource:0}: Error finding container c50506537a24c33c8181be8531fd13c7b9686325ab09dfa1dcb9ad484bf294cf: Status 404 returned error can't find the container with id c50506537a24c33c8181be8531fd13c7b9686325ab09dfa1dcb9ad484bf294cf
Sep 16 12:38:19.635867 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.635835    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.736467 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.736436    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.836627 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:19.836594    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-72.ec2.internal\" not found"
Sep 16 12:38:19.884896 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.884867    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:19.935039 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.935003    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:19.947862 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.947783    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:38:19.949030 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.949012    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal"
Sep 16 12:38:19.959532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:19.959512    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:38:20.310888 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.310791    2559 apiserver.go:52] "Watching apiserver"
Sep 16 12:38:20.318734 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.318709    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:38:20.320270 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.320244    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh","openshift-image-registry/node-ca-jxn2l","openshift-multus/multus-additional-cni-plugins-b69wk","openshift-multus/network-metrics-daemon-nc67r","openshift-network-operator/iptables-alerter-jdf72","openshift-ovn-kubernetes/ovnkube-node-j5sd5","kube-system/global-pull-secret-syncer-wn7dw","kube-system/konnectivity-agent-tjgsv","kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal","openshift-cluster-node-tuning-operator/tuned-t6tcj","openshift-dns/node-resolver-v47l6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal","openshift-multus/multus-xkn29","openshift-network-diagnostics/network-check-target-ckvgj"]
Sep 16 12:38:20.323837 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.323810    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.323940 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.323896    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:20.326341 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.326308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.328574 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.328554    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.330784 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.330572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.330784 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.330659    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:20.331441 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.331421    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:38:20.331768 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.331731    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-q7cjz\""
Sep 16 12:38:20.332531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.332322    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:38:20.332531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.332379    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:38:20.332531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.332379    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4bz7g\""
Sep 16 12:38:20.332531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.332525    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:38:20.334465 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.334438    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.334818 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.334799    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.335936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.335915    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:38:20.338710 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.338689    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.339224 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.339202    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.342268 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.342247    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:38:20.342382 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.342309    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.342434 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.342405    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:20.342482 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.342310    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4jq2w\""
Sep 16 12:38:20.342516 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.342478    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:20.343206 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.343186    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.345506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345455    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-cni-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.345506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345489    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-cnibin\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.345662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.345662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.345662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-cni-bin\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.345662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-netns\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.345662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345627    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-cni-bin\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.345662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-multus-certs\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345682    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjk9k\" (UniqueName: \"kubernetes.io/projected/9eca049f-d745-4785-b4a2-a85f698c8b3c-kube-api-access-zjk9k\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345709    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20815858-574f-437a-a3a4-641f9c0b5676-iptables-alerter-script\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345771    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20815858-574f-437a-a3a4-641f9c0b5676-host-slash\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345796    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovn-node-metrics-cert\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345812    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-ovn\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345828    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-env-overrides\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-system-cni-dir\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345859    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345875    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp98x\" (UniqueName: \"kubernetes.io/projected/0a7cb4e6-293e-433b-962e-6b2f052e9732-kube-api-access-fp98x\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.345960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345930    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjlvr\" (UniqueName: \"kubernetes.io/projected/20815858-574f-437a-a3a4-641f9c0b5676-kube-api-access-tjlvr\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.345992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-system-cni-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346029    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-socket-dir-parent\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-k8s-cni-cncf-io\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmnt\" (UniqueName: \"kubernetes.io/projected/07b7317e-46cb-4681-9614-e4185856b575-kube-api-access-csmnt\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346109    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-cni-netd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346123    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346132    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-cnibin\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-cni-multus\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346199    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-systemd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346222    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-var-lib-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346247    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-etc-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346286    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346373    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07b7317e-46cb-4681-9614-e4185856b575-cni-binary-copy\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-cni-binary-copy\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346454    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovnkube-script-lib\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.347801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346475    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-log-socket\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-kubelet\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-slash\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346568    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-run-netns\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346588    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-node-log\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswrd\" (UniqueName: \"kubernetes.io/projected/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-kube-api-access-vswrd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346622    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346633    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f74e27-80ff-462f-a926-9e533448687f-kubelet-config\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346649    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346739    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f74e27-80ff-462f-a926-9e533448687f-dbus\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-kubelet\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346819    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-hostroot\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346836    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-conf-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346862    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07b7317e-46cb-4681-9614-e4185856b575-multus-daemon-config\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346883    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-etc-kubernetes\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.346948    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-systemd-units\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.348679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.347001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.347056    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovnkube-config\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.347613    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.347694    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.347978    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.348407    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.349106    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:38:20.349553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.349292    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.349884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.349748    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k86ff\""
Sep 16 12:38:20.349884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.349770    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2scx4\""
Sep 16 12:38:20.349884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.349872    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.352425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.350910    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:38:20.352425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.351060    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.352425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.351548    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.352425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.352054    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.354181 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.354130    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-29zxv\""
Sep 16 12:38:20.354506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.354483    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:38:20.354618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.354592    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.354762 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.354746    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sllw2\""
Sep 16 12:38:20.354843 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.354826    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.355134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.355118    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.357768 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.357719    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.359310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.358283    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f5pl2\""
Sep 16 12:38:20.359310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.358311    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:38:20.359310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.358670    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:38:20.360317 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.360296    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:38:20.360651 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.360633    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4j7jj\""
Sep 16 12:38:20.361590 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.361570    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:38:20.367519 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.367497    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:38:20.399986 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.399941    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfx5p"
Sep 16 12:38:20.411887 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.411836    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal" event={"ID":"be62b2323e6968ec6b10000746a16003","Type":"ContainerStarted","Data":"c50506537a24c33c8181be8531fd13c7b9686325ab09dfa1dcb9ad484bf294cf"}
Sep 16 12:38:20.414000 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.413970    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal" event={"ID":"1fd67f18f3d247683da7ebab0e2c5954","Type":"ContainerStarted","Data":"58e40ac79cb23e994234d5f1fc961f912eb2e69f73931b26457f851065de7118"}
Sep 16 12:38:20.416929 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.416908    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jfx5p"
Sep 16 12:38:20.435910 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.435883    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:38:20.447841 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-netns\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.447980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447855    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-cni-bin\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.447980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c48de37d-b52d-44e7-bd6a-4515f7011d47-host\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.447980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447902    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-sys\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.447980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-env-overrides\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.447980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.447980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.447921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-netns\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448014    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-cni-bin\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448045    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp98x\" (UniqueName: \"kubernetes.io/projected/0a7cb4e6-293e-433b-962e-6b2f052e9732-kube-api-access-fp98x\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-kubernetes\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448113    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvtj9\" (UniqueName: \"kubernetes.io/projected/5d6a2d56-1f84-4af8-b22b-39db33e54337-kube-api-access-kvtj9\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448139    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-system-cni-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448163    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-k8s-cni-cncf-io\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmnt\" (UniqueName: \"kubernetes.io/projected/07b7317e-46cb-4681-9614-e4185856b575-kube-api-access-csmnt\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448204    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-tuning-conf-dir\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-sys-fs\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kxdk\" (UniqueName: \"kubernetes.io/projected/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-kube-api-access-4kxdk\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysconfig\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.448334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-cnibin\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448362    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-systemd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-etc-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07b7317e-46cb-4681-9614-e4185856b575-cni-binary-copy\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-cni-binary-copy\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-socket-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovnkube-script-lib\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448478    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-slash\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-env-overrides\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448556    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-run-netns\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-run-netns\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-node-log\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448614    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-systemd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswrd\" (UniqueName: \"kubernetes.io/projected/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-kube-api-access-vswrd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.448909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448648    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448673    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-k8s-cni-cncf-io\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5crst\" (UniqueName: \"kubernetes.io/projected/9828269d-2718-41c3-bc85-c03cbd535a49-kube-api-access-5crst\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448743    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-systemd-units\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448799    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovnkube-config\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20815858-574f-437a-a3a4-641f9c0b5676-iptables-alerter-script\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20815858-574f-437a-a3a4-641f9c0b5676-host-slash\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-cni-bin\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-etc-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-multus-certs\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448942    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysctl-conf\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-run-multus-certs\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20815858-574f-437a-a3a4-641f9c0b5676-host-slash\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.448998    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.449827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449033    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-systemd\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovn-node-metrics-cert\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-ovn\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449120    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-system-cni-dir\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449137    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-system-cni-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjlvr\" (UniqueName: \"kubernetes.io/projected/20815858-574f-437a-a3a4-641f9c0b5676-kube-api-access-tjlvr\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-cni-bin\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-socket-dir-parent\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-tmp-dir\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-lib-modules\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-var-lib-kubelet\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449289    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0739542-380d-4bdd-b027-d47a25b1645e-agent-certs\") pod \"konnectivity-agent-tjgsv\" (UID: \"b0739542-380d-4bdd-b027-d47a25b1645e\") " pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-cni-netd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449344    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-cni-multus\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-registration-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-hosts-file\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449485    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-modprobe-d\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.450744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449510    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-tuned\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449535    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-var-lib-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449594    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449660    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-log-socket\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449676    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449689    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-kubelet\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449717    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-kubelet\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-systemd-units\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-hostroot\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449774    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-device-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449799    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c48de37d-b52d-44e7-bd6a-4515f7011d47-serviceca\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449827    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6v8\" (UniqueName: \"kubernetes.io/projected/c48de37d-b52d-44e7-bd6a-4515f7011d47-kube-api-access-8l6v8\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f74e27-80ff-462f-a926-9e533448687f-kubelet-config\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449882    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f74e27-80ff-462f-a926-9e533448687f-dbus\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449903    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-cni-binary-copy\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-conf-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.451514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.449977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07b7317e-46cb-4681-9614-e4185856b575-multus-daemon-config\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-etc-kubernetes\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450058    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-run\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450083    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-host\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-cni-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450171    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovnkube-config\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450193    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-cnibin\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjk9k\" (UniqueName: \"kubernetes.io/projected/9eca049f-d745-4785-b4a2-a85f698c8b3c-kube-api-access-zjk9k\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysctl-d\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450296    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d6a2d56-1f84-4af8-b22b-39db33e54337-tmp\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450325    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0739542-380d-4bdd-b027-d47a25b1645e-konnectivity-ca\") pod \"konnectivity-agent-tjgsv\" (UID: \"b0739542-380d-4bdd-b027-d47a25b1645e\") " pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07b7317e-46cb-4681-9614-e4185856b575-cni-binary-copy\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.450471    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450497    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-cnibin\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450515    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovnkube-script-lib\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.450558    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:38:20.950519865 +0000 UTC m=+3.033962877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:20.452278 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0a7cb4e6-293e-433b-962e-6b2f052e9732-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450615    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-slash\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450658    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-node-log\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-kubelet\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450929    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-run-ovn\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450957    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-system-cni-dir\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.450988    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/20815858-574f-437a-a3a4-641f9c0b5676-iptables-alerter-script\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-kubelet\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-hostroot\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451148    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-run-ovn-kubernetes\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451193    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451256    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-cni-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451260    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-cni-netd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-host-var-lib-cni-multus\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451294    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-etc-kubernetes\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451288    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/07b7317e-46cb-4681-9614-e4185856b575-multus-daemon-config\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451349    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451396    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0a7cb4e6-293e-433b-962e-6b2f052e9732-cnibin\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.451438    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:20.452989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451446    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a0f74e27-80ff-462f-a926-9e533448687f-dbus\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.454091 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.451499    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:38:20.951481561 +0000 UTC m=+3.034924557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:20.454091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451499    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-var-lib-openvswitch\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.454091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451572    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a0f74e27-80ff-462f-a926-9e533448687f-kubelet-config\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.454091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451589    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-conf-dir\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.454091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/07b7317e-46cb-4681-9614-e4185856b575-multus-socket-dir-parent\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.454091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.451614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-log-socket\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.456193 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.456170    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-ovn-node-metrics-cert\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.463547 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.463519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmnt\" (UniqueName: \"kubernetes.io/projected/07b7317e-46cb-4681-9614-e4185856b575-kube-api-access-csmnt\") pod \"multus-xkn29\" (UID: \"07b7317e-46cb-4681-9614-e4185856b575\") " pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.464568 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.464518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswrd\" (UniqueName: \"kubernetes.io/projected/9cadeb8d-8b50-4ee0-8eca-078ffba1f845-kube-api-access-vswrd\") pod \"ovnkube-node-j5sd5\" (UID: \"9cadeb8d-8b50-4ee0-8eca-078ffba1f845\") " pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.464800 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.464781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp98x\" (UniqueName: \"kubernetes.io/projected/0a7cb4e6-293e-433b-962e-6b2f052e9732-kube-api-access-fp98x\") pod \"multus-additional-cni-plugins-b69wk\" (UID: \"0a7cb4e6-293e-433b-962e-6b2f052e9732\") " pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.465207 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.465189    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjk9k\" (UniqueName: \"kubernetes.io/projected/9eca049f-d745-4785-b4a2-a85f698c8b3c-kube-api-access-zjk9k\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.466945 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.466919    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjlvr\" (UniqueName: \"kubernetes.io/projected/20815858-574f-437a-a3a4-641f9c0b5676-kube-api-access-tjlvr\") pod \"iptables-alerter-jdf72\" (UID: \"20815858-574f-437a-a3a4-641f9c0b5676\") " pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.551567 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-sys-fs\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.551567 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kxdk\" (UniqueName: \"kubernetes.io/projected/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-kube-api-access-4kxdk\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysconfig\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-socket-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551656    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5crst\" (UniqueName: \"kubernetes.io/projected/9828269d-2718-41c3-bc85-c03cbd535a49-kube-api-access-5crst\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-sys-fs\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysctl-conf\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551752    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-systemd\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551778    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-tmp-dir\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551817    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-etc-selinux\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.551834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551830    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysconfig\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-lib-modules\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-var-lib-kubelet\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551926    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0739542-380d-4bdd-b027-d47a25b1645e-agent-certs\") pod \"konnectivity-agent-tjgsv\" (UID: \"b0739542-380d-4bdd-b027-d47a25b1645e\") " pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-registration-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-hosts-file\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552006    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-modprobe-d\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551995    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysctl-conf\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.551962    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-socket-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552032    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-lib-modules\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552058    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-systemd\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-hosts-file\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552088    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-tmp-dir\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552095    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-tuned\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-registration-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552197    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-modprobe-d\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552231    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.552408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552268    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-device-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552278    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-var-lib-kubelet\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c48de37d-b52d-44e7-bd6a-4515f7011d47-serviceca\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552320    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6v8\" (UniqueName: \"kubernetes.io/projected/c48de37d-b52d-44e7-bd6a-4515f7011d47-kube-api-access-8l6v8\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552356    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552360    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9828269d-2718-41c3-bc85-c03cbd535a49-device-dir\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-run\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-host\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysctl-d\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552528    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d6a2d56-1f84-4af8-b22b-39db33e54337-tmp\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552551    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0739542-380d-4bdd-b027-d47a25b1645e-konnectivity-ca\") pod \"konnectivity-agent-tjgsv\" (UID: \"b0739542-380d-4bdd-b027-d47a25b1645e\") " pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c48de37d-b52d-44e7-bd6a-4515f7011d47-host\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-sys\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-kubernetes\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-sysctl-d\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552668    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvtj9\" (UniqueName: \"kubernetes.io/projected/5d6a2d56-1f84-4af8-b22b-39db33e54337-kube-api-access-kvtj9\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552771    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-run\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-host\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.552979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c48de37d-b52d-44e7-bd6a-4515f7011d47-serviceca\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.553998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.553047    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c48de37d-b52d-44e7-bd6a-4515f7011d47-host\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.553998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.553359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-sys\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.553436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-kubernetes\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.553998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.553568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b0739542-380d-4bdd-b027-d47a25b1645e-konnectivity-ca\") pod \"konnectivity-agent-tjgsv\" (UID: \"b0739542-380d-4bdd-b027-d47a25b1645e\") " pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.554694 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.554675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d6a2d56-1f84-4af8-b22b-39db33e54337-etc-tuned\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.555062 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.555044    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d6a2d56-1f84-4af8-b22b-39db33e54337-tmp\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.555185 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.555167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b0739542-380d-4bdd-b027-d47a25b1645e-agent-certs\") pod \"konnectivity-agent-tjgsv\" (UID: \"b0739542-380d-4bdd-b027-d47a25b1645e\") " pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.563027 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.562553    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:20.563027 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.562580    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:20.563027 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.562594    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:20.563027 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.562659    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:21.06263969 +0000 UTC m=+3.146082687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:20.565643 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.565617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvtj9\" (UniqueName: \"kubernetes.io/projected/5d6a2d56-1f84-4af8-b22b-39db33e54337-kube-api-access-kvtj9\") pod \"tuned-t6tcj\" (UID: \"5d6a2d56-1f84-4af8-b22b-39db33e54337\") " pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.566427 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.566376    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5crst\" (UniqueName: \"kubernetes.io/projected/9828269d-2718-41c3-bc85-c03cbd535a49-kube-api-access-5crst\") pod \"aws-ebs-csi-driver-node-x7snh\" (UID: \"9828269d-2718-41c3-bc85-c03cbd535a49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.567574 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.567538    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kxdk\" (UniqueName: \"kubernetes.io/projected/9fb924aa-f0e1-46e9-97d6-e5db3cdd500c-kube-api-access-4kxdk\") pod \"node-resolver-v47l6\" (UID: \"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c\") " pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.568562 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.568525    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6v8\" (UniqueName: \"kubernetes.io/projected/c48de37d-b52d-44e7-bd6a-4515f7011d47-kube-api-access-8l6v8\") pod \"node-ca-jxn2l\" (UID: \"c48de37d-b52d-44e7-bd6a-4515f7011d47\") " pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.641838 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.641795    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xkn29"
Sep 16 12:38:20.651685 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.651659    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-b69wk"
Sep 16 12:38:20.663472 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.663445    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jdf72"
Sep 16 12:38:20.668250 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.668229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:20.683131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.683102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh"
Sep 16 12:38:20.694081 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.694057    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jxn2l"
Sep 16 12:38:20.705025 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.705007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t6tcj"
Sep 16 12:38:20.712852 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.712834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v47l6"
Sep 16 12:38:20.720848 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.720831    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:20.956944 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.956871    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:20.956944 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:20.956919    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:20.957115 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.957027    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:20.957115 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.957028    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:20.957115 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.957082    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:38:21.95706896 +0000 UTC m=+4.040511954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:20.957115 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:20.957097    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:38:21.957089989 +0000 UTC m=+4.040532983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:21.158473 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:21.158430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:21.158655 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.158586    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:21.158655 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.158618    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:21.158655 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.158629    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:21.158811 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.158694    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:22.158676921 +0000 UTC m=+4.242119915 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:21.421159 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:21.421050    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:33:21 +0000 UTC" deadline="2028-02-10 09:57:58.26386863 +0000 UTC"
Sep 16 12:38:21.421159 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:21.421092    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12285h19m36.842781341s"
Sep 16 12:38:21.733816 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.733786    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b7317e_46cb_4681_9614_e4185856b575.slice/crio-cb17b5072e707e78250f21a30321345181eba27b49aea7c69885f433a7177bd2 WatchSource:0}: Error finding container cb17b5072e707e78250f21a30321345181eba27b49aea7c69885f433a7177bd2: Status 404 returned error can't find the container with id cb17b5072e707e78250f21a30321345181eba27b49aea7c69885f433a7177bd2
Sep 16 12:38:21.735132 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.735091    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb924aa_f0e1_46e9_97d6_e5db3cdd500c.slice/crio-728d298f2a428af12451aa0d30bf38f2488cde420f337a92bd989a7af69e04df WatchSource:0}: Error finding container 728d298f2a428af12451aa0d30bf38f2488cde420f337a92bd989a7af69e04df: Status 404 returned error can't find the container with id 728d298f2a428af12451aa0d30bf38f2488cde420f337a92bd989a7af69e04df
Sep 16 12:38:21.737178 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.737091    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9828269d_2718_41c3_bc85_c03cbd535a49.slice/crio-d03e0b057fb1e3d61cb72e72fd9e15133dffd1914b36a04584195bff96082464 WatchSource:0}: Error finding container d03e0b057fb1e3d61cb72e72fd9e15133dffd1914b36a04584195bff96082464: Status 404 returned error can't find the container with id d03e0b057fb1e3d61cb72e72fd9e15133dffd1914b36a04584195bff96082464
Sep 16 12:38:21.742033 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.741903    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48de37d_b52d_44e7_bd6a_4515f7011d47.slice/crio-ff94339953f21a7cd1ec8df2d6ca6eeea05be0801762524ca41b11853c713d85 WatchSource:0}: Error finding container ff94339953f21a7cd1ec8df2d6ca6eeea05be0801762524ca41b11853c713d85: Status 404 returned error can't find the container with id ff94339953f21a7cd1ec8df2d6ca6eeea05be0801762524ca41b11853c713d85
Sep 16 12:38:21.743688 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.743544    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6a2d56_1f84_4af8_b22b_39db33e54337.slice/crio-e729685d33fbff08fcd6d9e111901b926fffd66ae8be10ebad0543ad5b6b4a26 WatchSource:0}: Error finding container e729685d33fbff08fcd6d9e111901b926fffd66ae8be10ebad0543ad5b6b4a26: Status 404 returned error can't find the container with id e729685d33fbff08fcd6d9e111901b926fffd66ae8be10ebad0543ad5b6b4a26
Sep 16 12:38:21.744792 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.744260    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20815858_574f_437a_a3a4_641f9c0b5676.slice/crio-700db50688bf4b6cc2a2f76c6f6adbde33c9b7aa5aa01ee841f8ab5823ad887e WatchSource:0}: Error finding container 700db50688bf4b6cc2a2f76c6f6adbde33c9b7aa5aa01ee841f8ab5823ad887e: Status 404 returned error can't find the container with id 700db50688bf4b6cc2a2f76c6f6adbde33c9b7aa5aa01ee841f8ab5823ad887e
Sep 16 12:38:21.745714 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.745686    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7cb4e6_293e_433b_962e_6b2f052e9732.slice/crio-e30aa9a62951110c6e818a33b65ae7e68dc1e9fbf1ad958e381f3c30b504ee9d WatchSource:0}: Error finding container e30aa9a62951110c6e818a33b65ae7e68dc1e9fbf1ad958e381f3c30b504ee9d: Status 404 returned error can't find the container with id e30aa9a62951110c6e818a33b65ae7e68dc1e9fbf1ad958e381f3c30b504ee9d
Sep 16 12:38:21.746459 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.746332    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cadeb8d_8b50_4ee0_8eca_078ffba1f845.slice/crio-39966b33f7051edfd63ac0cd5d91617107469459356028a3dcb3e41fe681efb3 WatchSource:0}: Error finding container 39966b33f7051edfd63ac0cd5d91617107469459356028a3dcb3e41fe681efb3: Status 404 returned error can't find the container with id 39966b33f7051edfd63ac0cd5d91617107469459356028a3dcb3e41fe681efb3
Sep 16 12:38:21.747870 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:21.747845    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0739542_380d_4bdd_b027_d47a25b1645e.slice/crio-a625f5602716bd49616b85e1b69ad225ee807e274deceeef8b95aff0383c3a4e WatchSource:0}: Error finding container a625f5602716bd49616b85e1b69ad225ee807e274deceeef8b95aff0383c3a4e: Status 404 returned error can't find the container with id a625f5602716bd49616b85e1b69ad225ee807e274deceeef8b95aff0383c3a4e
Sep 16 12:38:21.964373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:21.964319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:21.964554 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:21.964441    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:21.964554 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.964513    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:21.964682 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.964554    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:21.964682 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.964591    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:38:23.964575341 +0000 UTC m=+6.048018345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:21.964682 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:21.964610    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:38:23.964603223 +0000 UTC m=+6.048046217 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:22.165693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.165655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:22.166097 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.165871    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:22.166097 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.165902    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:22.166097 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.165917    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:22.166097 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.165971    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:24.165956796 +0000 UTC m=+6.249399790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:22.407961 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.406962    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:22.407961 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.407096    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:22.407961 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.407576    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:22.407961 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.407677    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:22.407961 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.407785    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:22.407961 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:22.407879    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:22.421624 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.421587    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 12:33:21 +0000 UTC" deadline="2028-03-17 20:24:34.78820902 +0000 UTC"
Sep 16 12:38:22.421624 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.421623    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13159h46m12.366590687s"
Sep 16 12:38:22.425799 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.425769    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal" event={"ID":"be62b2323e6968ec6b10000746a16003","Type":"ContainerStarted","Data":"8e067352f72da2c18105cfa03d8fa69fda257f69367829190dc0ba3c90d1d658"}
Sep 16 12:38:22.441434 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.440764    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-72.ec2.internal" podStartSLOduration=2.440743888 podStartE2EDuration="2.440743888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:22.440719539 +0000 UTC m=+4.524162567" watchObservedRunningTime="2026-09-16 12:38:22.440743888 +0000 UTC m=+4.524186908"
Sep 16 12:38:22.448582 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.447874    2559 generic.go:356] "Generic (PLEG): container finished" podID="1fd67f18f3d247683da7ebab0e2c5954" containerID="9d5b3969f70bd5806e1ddc700775aabdbcdcedab2f12235d897a70c2da40867c" exitCode=0
Sep 16 12:38:22.448582 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.447969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal" event={"ID":"1fd67f18f3d247683da7ebab0e2c5954","Type":"ContainerDied","Data":"9d5b3969f70bd5806e1ddc700775aabdbcdcedab2f12235d897a70c2da40867c"}
Sep 16 12:38:22.456515 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.456478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjgsv" event={"ID":"b0739542-380d-4bdd-b027-d47a25b1645e","Type":"ContainerStarted","Data":"a625f5602716bd49616b85e1b69ad225ee807e274deceeef8b95aff0383c3a4e"}
Sep 16 12:38:22.473609 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.473554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"39966b33f7051edfd63ac0cd5d91617107469459356028a3dcb3e41fe681efb3"}
Sep 16 12:38:22.485074 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.484971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh" event={"ID":"9828269d-2718-41c3-bc85-c03cbd535a49","Type":"ContainerStarted","Data":"d03e0b057fb1e3d61cb72e72fd9e15133dffd1914b36a04584195bff96082464"}
Sep 16 12:38:22.494319 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.494074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xkn29" event={"ID":"07b7317e-46cb-4681-9614-e4185856b575","Type":"ContainerStarted","Data":"cb17b5072e707e78250f21a30321345181eba27b49aea7c69885f433a7177bd2"}
Sep 16 12:38:22.502244 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.502210    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdf72" event={"ID":"20815858-574f-437a-a3a4-641f9c0b5676","Type":"ContainerStarted","Data":"700db50688bf4b6cc2a2f76c6f6adbde33c9b7aa5aa01ee841f8ab5823ad887e"}
Sep 16 12:38:22.529443 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.529361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerStarted","Data":"e30aa9a62951110c6e818a33b65ae7e68dc1e9fbf1ad958e381f3c30b504ee9d"}
Sep 16 12:38:22.543758 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.543645    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6tcj" event={"ID":"5d6a2d56-1f84-4af8-b22b-39db33e54337","Type":"ContainerStarted","Data":"e729685d33fbff08fcd6d9e111901b926fffd66ae8be10ebad0543ad5b6b4a26"}
Sep 16 12:38:22.565658 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.565612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxn2l" event={"ID":"c48de37d-b52d-44e7-bd6a-4515f7011d47","Type":"ContainerStarted","Data":"ff94339953f21a7cd1ec8df2d6ca6eeea05be0801762524ca41b11853c713d85"}
Sep 16 12:38:22.573877 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:22.573827    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v47l6" event={"ID":"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c","Type":"ContainerStarted","Data":"728d298f2a428af12451aa0d30bf38f2488cde420f337a92bd989a7af69e04df"}
Sep 16 12:38:23.604908 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:23.604774    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal" event={"ID":"1fd67f18f3d247683da7ebab0e2c5954","Type":"ContainerStarted","Data":"43ee3e723b699b794f756946fe4ab111e9c9124e265c706aaf4841ca1766a2e8"}
Sep 16 12:38:23.981887 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:23.981792    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:23.981887 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:23.981868    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:23.982123 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:23.982014    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:23.982177 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:23.982125    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:38:27.982059864 +0000 UTC m=+10.065502874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:23.982664 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:23.982640    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:23.982797 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:23.982700    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:38:27.982684094 +0000 UTC m=+10.066127103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:24.183222 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:24.183179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:24.183449 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.183388    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:24.183449 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.183413    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:24.183449 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.183426    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:24.183634 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.183494    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:28.183472229 +0000 UTC m=+10.266915230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:24.408128 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:24.408033    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:24.408387 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.408190    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:24.408631 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:24.408609    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:24.408754 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.408712    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:24.408836 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:24.408769    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:24.408888 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:24.408836    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:26.409264 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:26.408529    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:26.409264 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:26.408669    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:26.409264 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:26.409005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:26.409264 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:26.409089    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:26.409264 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:26.409133    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:26.409264 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:26.409180    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:28.017312 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:28.017272    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:28.017827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:28.017368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:28.017827 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.017568    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:28.017827 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.017640    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:38:36.017621139 +0000 UTC m=+18.101064152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:28.017827 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.017567    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:28.017827 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.017708    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:38:36.017692006 +0000 UTC m=+18.101135000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:28.219461 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:28.219420    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:28.219670 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.219630    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:28.219670 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.219659    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:28.219787 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.219674    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:28.219787 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.219746    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:36.219717361 +0000 UTC m=+18.303160361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:28.407201 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:28.406489    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:28.407201 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.406737    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:28.407201 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:28.406783    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:28.407201 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.406923    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:28.407201 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:28.406984    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:28.407201 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:28.407051    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:30.406661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:30.405980    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:30.406661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:30.406002    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:30.406661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:30.405980    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:30.406661 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:30.406114    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:30.406661 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:30.406537    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:30.406661 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:30.406622    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:32.406008 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:32.405964    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:32.406008 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:32.406002    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:32.406518 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:32.405970    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:32.406518 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:32.406102    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:32.406518 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:32.406171    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:32.406838 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:32.406769    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:34.406235 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:34.405740    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:34.406235 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:34.405740    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:34.406235 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:34.405874    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:34.406235 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:34.405869    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:34.406235 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:34.405943    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:34.406235 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:34.406045    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:36.083346 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:36.083230    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:36.083346 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:36.083339    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:36.083806 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.083419    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:36.083806 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.083453    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:36.083806 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.083506    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:38:52.083482314 +0000 UTC m=+34.166925513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:36.083806 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.083526    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:38:52.083517161 +0000 UTC m=+34.166960155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:36.284951 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:36.284910    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:36.285143 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.285116    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:36.285143 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.285148    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:36.285305 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.285161    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:36.285305 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.285232    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:52.285213 +0000 UTC m=+34.368656007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:36.410334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:36.409804    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:36.410334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:36.409913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:36.410334 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:36.409927    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:36.410334 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.409927    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:36.410334 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.410022    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:36.410334 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:36.410095    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:38.409264 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:38.409235    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:38.409740 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:38.409372    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:38.409740 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:38.409382    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:38.409740 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:38.409419    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:38.409740 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:38.409513    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:38.409740 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:38.409604    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:40.405868 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.405635    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:40.406616 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.405646    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:40.406616 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:40.405981    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:40.406616 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:40.406050    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:40.406616 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.405662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:40.406616 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:40.406131    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:40.640580 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.640553    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh" event={"ID":"9828269d-2718-41c3-bc85-c03cbd535a49","Type":"ContainerStarted","Data":"85a4f456c347f08b8c45d46e72655d014ccb2b3adf7b1c425f098ecd9d6f188a"}
Sep 16 12:38:40.641993 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.641962    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xkn29" event={"ID":"07b7317e-46cb-4681-9614-e4185856b575","Type":"ContainerStarted","Data":"f5479734e06283bc64d5d3120c78185b9e6068b77bcba08a138b9e2f2b6f35f3"}
Sep 16 12:38:40.645744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.645535    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7cb4e6-293e-433b-962e-6b2f052e9732" containerID="e1caf753bd16b8bb006ce785974659342aa6c95d8a97a403c2c004a39bc907e8" exitCode=0
Sep 16 12:38:40.645838 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.645606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerDied","Data":"e1caf753bd16b8bb006ce785974659342aa6c95d8a97a403c2c004a39bc907e8"}
Sep 16 12:38:40.647420 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.647396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t6tcj" event={"ID":"5d6a2d56-1f84-4af8-b22b-39db33e54337","Type":"ContainerStarted","Data":"4652d54c3560ba3c3fe8efcf37e063b2f4011aa164a885d40157ae514323b3f2"}
Sep 16 12:38:40.650226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.649039    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jxn2l" event={"ID":"c48de37d-b52d-44e7-bd6a-4515f7011d47","Type":"ContainerStarted","Data":"f88af17a47bc37411efe192b3b9c386364ff2335390df96e09dcc85e97ac9468"}
Sep 16 12:38:40.650784 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.650760    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v47l6" event={"ID":"9fb924aa-f0e1-46e9-97d6-e5db3cdd500c","Type":"ContainerStarted","Data":"97a492bdc7d53fa23e0ae4790046dd155b6104ee580ac5f5b1c03ab744392b39"}
Sep 16 12:38:40.652401 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.652376    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tjgsv" event={"ID":"b0739542-380d-4bdd-b027-d47a25b1645e","Type":"ContainerStarted","Data":"7cd4fc2e9d93ec898d2b8523a306ac1a658a3cb369c6de2ea9fc634741863f41"}
Sep 16 12:38:40.655723 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.655700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"de3def1f84952afbd00f61f491dd4a5fe200a978394061d1c59880ddf014afbb"}
Sep 16 12:38:40.655812 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.655777    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"4ae70789a908041a52ae629b044acaf5f585f66bc950c01713f262d89c3a55a6"}
Sep 16 12:38:40.655856 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.655828    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"aa96935ab054adc27a7749c2f764cdb2f8c920c740fd88167fee8d0d73d65e9b"}
Sep 16 12:38:40.655886 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.655877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"335ccb302c2c75f5d8321fa5b086c4edb037705b76b7aff3c7582a3f709a4003"}
Sep 16 12:38:40.655946 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.655933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"d658a34dfe1f87c05bd78d29952ef284936e83cc34361386ab5b12fc0fc8f7eb"}
Sep 16 12:38:40.655996 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.655984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"933a35c460eb7ed4b95fb856e32c92ed14ef5831f960db67596459286ce50048"}
Sep 16 12:38:40.658556 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.658513    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-72.ec2.internal" podStartSLOduration=20.658501673 podStartE2EDuration="20.658501673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:23.621928481 +0000 UTC m=+5.705371500" watchObservedRunningTime="2026-09-16 12:38:40.658501673 +0000 UTC m=+22.741944688"
Sep 16 12:38:40.658656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.658606    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xkn29" podStartSLOduration=3.71948649 podStartE2EDuration="21.658597118s" totalImagesPullingTime="17.939110628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:40.658120964 +0000 UTC m=+22.741564002" watchObservedRunningTime="2026-09-16 12:38:40.658597118 +0000 UTC m=+22.742040147"
Sep 16 12:38:40.671937 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.671898    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jxn2l" podStartSLOduration=3.892843448 podStartE2EDuration="21.671886227s" totalImagesPullingTime="17.779042779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:40.671574338 +0000 UTC m=+22.755017371" watchObservedRunningTime="2026-09-16 12:38:40.671886227 +0000 UTC m=+22.755329241"
Sep 16 12:38:40.705249 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.705204    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t6tcj" podStartSLOduration=3.904285488 podStartE2EDuration="21.70519003s" totalImagesPullingTime="17.800904542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:40.688568075 +0000 UTC m=+22.772011100" watchObservedRunningTime="2026-09-16 12:38:40.70519003 +0000 UTC m=+22.788633044"
Sep 16 12:38:40.719602 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.719545    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tjgsv" podStartSLOduration=3.94774295 podStartE2EDuration="21.719524276s" totalImagesPullingTime="17.771781326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:40.719151722 +0000 UTC m=+22.802594747" watchObservedRunningTime="2026-09-16 12:38:40.719524276 +0000 UTC m=+22.802967290"
Sep 16 12:38:40.719812 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.719778    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v47l6" podStartSLOduration=3.910769784 podStartE2EDuration="21.719770164s" totalImagesPullingTime="17.80900038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:40.704653785 +0000 UTC m=+22.788096809" watchObservedRunningTime="2026-09-16 12:38:40.719770164 +0000 UTC m=+22.803213182"
Sep 16 12:38:40.729226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:40.729205    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:38:41.410864 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:41.410708    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:38:40.729222752Z","UUID":"ed550a66-c3f0-4195-bc1d-e499e8cf51e4","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:38:41.412598 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:41.412567    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:38:41.412696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:41.412604    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:38:41.660402 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:41.660362    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jdf72" event={"ID":"20815858-574f-437a-a3a4-641f9c0b5676","Type":"ContainerStarted","Data":"a74d1da2f372fc9dd1ecc0cfa8663204db6f698b2c8c29888d1d3bf7335d0d0c"}
Sep 16 12:38:41.662770 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:41.662694    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh" event={"ID":"9828269d-2718-41c3-bc85-c03cbd535a49","Type":"ContainerStarted","Data":"5ff6016149a6254a798b30dd3a0cd4a4110c06f15001d838ae7df3d6037733e3"}
Sep 16 12:38:41.679225 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:41.679168    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jdf72" podStartSLOduration=4.894237481 podStartE2EDuration="22.679151788s" totalImagesPullingTime="17.784914307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:41.678685572 +0000 UTC m=+23.762128623" watchObservedRunningTime="2026-09-16 12:38:41.679151788 +0000 UTC m=+23.762594802"
Sep 16 12:38:42.409198 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.409171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:42.409390 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.409212    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:42.409390 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.409171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:42.409390 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:42.409297    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:42.409390 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:42.409371    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:42.409607 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:42.409455    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:42.670805 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.670701    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"0af0fc74b7c9c08e4fa522dc9e51b6285834c53b940ef008540e623f4c0bdf43"}
Sep 16 12:38:42.673166 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.673097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh" event={"ID":"9828269d-2718-41c3-bc85-c03cbd535a49","Type":"ContainerStarted","Data":"a20e603cf3f7d0e8f2b899288c99a26ec5ce79ae8989fa671a2924ee18d6c8ec"}
Sep 16 12:38:42.694560 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.694515    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x7snh" podStartSLOduration=3.836885348 podStartE2EDuration="23.694499754s" totalImagesPullingTime="19.857614406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:42.694473067 +0000 UTC m=+24.777916080" watchObservedRunningTime="2026-09-16 12:38:42.694499754 +0000 UTC m=+24.777942745"
Sep 16 12:38:42.960905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.960819    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:42.961526 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:42.961501    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:43.675725 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:43.675688    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:43.676129 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:43.676061    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tjgsv"
Sep 16 12:38:44.405920 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:44.405890    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:44.405920 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:44.405929    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:44.406159 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:44.406007    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:44.406200 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:44.406147    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:44.406426 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:44.406241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:44.406426 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:44.406387    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:45.683901 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.683729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" event={"ID":"9cadeb8d-8b50-4ee0-8eca-078ffba1f845","Type":"ContainerStarted","Data":"f08cd6a3accea29b3b917c8c77ae54085ce94c5c1f6797042a359fc00902176f"}
Sep 16 12:38:45.683901 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.683971    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:45.684605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.684046    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:45.684605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.684093    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:45.699052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.699032    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:45.699149 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.699129    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:38:45.716498 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:45.715117    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5" podStartSLOduration=8.885725485 podStartE2EDuration="26.715091479s" totalImagesPullingTime="17.829365994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:45.714575558 +0000 UTC m=+27.798018568" watchObservedRunningTime="2026-09-16 12:38:45.715091479 +0000 UTC m=+27.798534497"
Sep 16 12:38:46.406137 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:46.406108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:46.406344 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:46.406108    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:46.406344 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:46.406205    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:46.406344 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:46.406241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:46.406454 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:46.406348    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:46.406454 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:46.406419    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:46.688461 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:46.688433    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7cb4e6-293e-433b-962e-6b2f052e9732" containerID="6b9bc175d124047e35491ee6db6ae4ad50767e5048275f1df58a6ffdedcc5c1a" exitCode=0
Sep 16 12:38:46.688934 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:46.688514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerDied","Data":"6b9bc175d124047e35491ee6db6ae4ad50767e5048275f1df58a6ffdedcc5c1a"}
Sep 16 12:38:47.693122 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.692873    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7cb4e6-293e-433b-962e-6b2f052e9732" containerID="ffca8868afb2738d685d3a0596d414eff5f600c0c55039276ff7ffd1ff7d4eee" exitCode=0
Sep 16 12:38:47.693512 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.692935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerDied","Data":"ffca8868afb2738d685d3a0596d414eff5f600c0c55039276ff7ffd1ff7d4eee"}
Sep 16 12:38:47.723590 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.723558    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wn7dw"]
Sep 16 12:38:47.723739 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.723720    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:47.723874 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:47.723845    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:47.727310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.727284    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ckvgj"]
Sep 16 12:38:47.727459 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.727398    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:47.727524 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:47.727489    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:47.727861 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.727829    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nc67r"]
Sep 16 12:38:47.727959 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:47.727934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:47.728082 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:47.728048    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:49.405421 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:49.405387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:49.405834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:49.405429    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:49.405834 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:49.405387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:49.405834 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:49.405499    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:49.405834 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:49.405584    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:49.405834 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:49.405658    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:49.699422 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:49.699309    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7cb4e6-293e-433b-962e-6b2f052e9732" containerID="6ebf653b3fe70e737851e7a610f871de160fbc42d3d3b2dcefa7b0bbb4722500" exitCode=0
Sep 16 12:38:49.699422 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:49.699398    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerDied","Data":"6ebf653b3fe70e737851e7a610f871de160fbc42d3d3b2dcefa7b0bbb4722500"}
Sep 16 12:38:51.406250 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:51.406210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:51.406790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:51.406210    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:51.406790 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:51.406378    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wn7dw" podUID="a0f74e27-80ff-462f-a926-9e533448687f"
Sep 16 12:38:51.406790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:51.406225    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:51.406790 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:51.406455    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:38:51.406790 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:51.406499    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ckvgj" podUID="c9ec9a45-d9df-439f-b42e-ff9a0f289684"
Sep 16 12:38:52.102908 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.102872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:52.103096 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.102938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:52.103096 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.103031    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:52.103096 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.103031    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:52.103096 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.103098    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:39:24.103078416 +0000 UTC m=+66.186521410 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:38:52.103574 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.103115    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret podName:a0f74e27-80ff-462f-a926-9e533448687f nodeName:}" failed. No retries permitted until 2026-09-16 12:39:24.103107212 +0000 UTC m=+66.186550204 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret") pod "global-pull-secret-syncer-wn7dw" (UID: "a0f74e27-80ff-462f-a926-9e533448687f") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:38:52.253797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.253770    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-72.ec2.internal" event="NodeReady"
Sep 16 12:38:52.253980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.253908    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:38:52.304985 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.304949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:52.305161 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.305112    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:38:52.305161 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.305136    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:38:52.305161 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.305149    2559 projected.go:196] Error preparing data for projected volume kube-api-access-28848 for pod openshift-network-diagnostics/network-check-target-ckvgj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:52.305288 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.305204    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848 podName:c9ec9a45-d9df-439f-b42e-ff9a0f289684 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:24.305187848 +0000 UTC m=+66.388630853 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-28848" (UniqueName: "kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848") pod "network-check-target-ckvgj" (UID: "c9ec9a45-d9df-439f-b42e-ff9a0f289684") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:38:52.313201 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.312960    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vls4q"]
Sep 16 12:38:52.334233 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.334201    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zsmv"]
Sep 16 12:38:52.334406 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.334361    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.339082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.339062    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:38:52.339243 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.339224    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:38:52.339312 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.339231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fgm9v\""
Sep 16 12:38:52.350265 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.350243    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vls4q"]
Sep 16 12:38:52.350413 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.350282    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zsmv"]
Sep 16 12:38:52.350413 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.350294    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.353344 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.353268    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:38:52.353452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.353406    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g4k4m\""
Sep 16 12:38:52.353515 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.353503    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:38:52.354000 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.353969    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:38:52.405667 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.405631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.405667 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.405673    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60881a7a-95f1-4713-bddd-7e0e36b2e403-config-volume\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.405936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.405755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkkh9\" (UniqueName: \"kubernetes.io/projected/60881a7a-95f1-4713-bddd-7e0e36b2e403-kube-api-access-dkkh9\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.405936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.405863    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60881a7a-95f1-4713-bddd-7e0e36b2e403-tmp-dir\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.506981 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.506949    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aca5a17-6f82-43f7-9b53-c80428d48ac2-cert\") pod \"ingress-canary-8zsmv\" (UID: \"1aca5a17-6f82-43f7-9b53-c80428d48ac2\") " pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.506981 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.506986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th87f\" (UniqueName: \"kubernetes.io/projected/1aca5a17-6f82-43f7-9b53-c80428d48ac2-kube-api-access-th87f\") pod \"ingress-canary-8zsmv\" (UID: \"1aca5a17-6f82-43f7-9b53-c80428d48ac2\") " pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.507471 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.507092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60881a7a-95f1-4713-bddd-7e0e36b2e403-tmp-dir\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.507471 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.507144    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.507471 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.507162    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60881a7a-95f1-4713-bddd-7e0e36b2e403-config-volume\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.507471 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.507187    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkkh9\" (UniqueName: \"kubernetes.io/projected/60881a7a-95f1-4713-bddd-7e0e36b2e403-kube-api-access-dkkh9\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.507471 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.507287    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:52.507471 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:52.507366    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:53.007350191 +0000 UTC m=+35.090793187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:52.507748 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.507703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60881a7a-95f1-4713-bddd-7e0e36b2e403-tmp-dir\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.507841 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.507821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/60881a7a-95f1-4713-bddd-7e0e36b2e403-config-volume\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.518073 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.517943    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkkh9\" (UniqueName: \"kubernetes.io/projected/60881a7a-95f1-4713-bddd-7e0e36b2e403-kube-api-access-dkkh9\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:52.608190 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.608106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aca5a17-6f82-43f7-9b53-c80428d48ac2-cert\") pod \"ingress-canary-8zsmv\" (UID: \"1aca5a17-6f82-43f7-9b53-c80428d48ac2\") " pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.608190 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.608148    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th87f\" (UniqueName: \"kubernetes.io/projected/1aca5a17-6f82-43f7-9b53-c80428d48ac2-kube-api-access-th87f\") pod \"ingress-canary-8zsmv\" (UID: \"1aca5a17-6f82-43f7-9b53-c80428d48ac2\") " pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.611066 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.611034    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1aca5a17-6f82-43f7-9b53-c80428d48ac2-cert\") pod \"ingress-canary-8zsmv\" (UID: \"1aca5a17-6f82-43f7-9b53-c80428d48ac2\") " pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.617656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.617628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th87f\" (UniqueName: \"kubernetes.io/projected/1aca5a17-6f82-43f7-9b53-c80428d48ac2-kube-api-access-th87f\") pod \"ingress-canary-8zsmv\" (UID: \"1aca5a17-6f82-43f7-9b53-c80428d48ac2\") " pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:52.660593 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:52.660557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zsmv"
Sep 16 12:38:53.011015 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.010925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:53.011202 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:53.011087    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:53.011202 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:53.011154    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:54.011136921 +0000 UTC m=+36.094579912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:53.405955 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.405863    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:38:53.405955 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.405905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:38:53.405955 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.405905    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:38:53.408826 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.408805    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:38:53.409012 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.408998    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:38:53.409122 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.409108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:38:53.409629 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.409435    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:38:53.409629 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.409435    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x7vdr\""
Sep 16 12:38:53.409629 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:53.409601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b66st\""
Sep 16 12:38:54.018654 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:54.018619    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:54.019084 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:54.018781    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:54.019084 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:54.018845    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:38:56.018829735 +0000 UTC m=+38.102272740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:54.482632 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:54.482567    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zsmv"]
Sep 16 12:38:54.486362 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:38:54.486297    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aca5a17_6f82_43f7_9b53_c80428d48ac2.slice/crio-21123f1e2f46dea8ac9d6f5320de0c26d59e998bd100bd18ea411fd63c9c78db WatchSource:0}: Error finding container 21123f1e2f46dea8ac9d6f5320de0c26d59e998bd100bd18ea411fd63c9c78db: Status 404 returned error can't find the container with id 21123f1e2f46dea8ac9d6f5320de0c26d59e998bd100bd18ea411fd63c9c78db
Sep 16 12:38:54.712860 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:54.712819    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zsmv" event={"ID":"1aca5a17-6f82-43f7-9b53-c80428d48ac2","Type":"ContainerStarted","Data":"21123f1e2f46dea8ac9d6f5320de0c26d59e998bd100bd18ea411fd63c9c78db"}
Sep 16 12:38:54.715523 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:54.715499    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7cb4e6-293e-433b-962e-6b2f052e9732" containerID="7852f05f6727101612d7a30a9066c4e5d30a6fae48c6534765912ca7639705db" exitCode=0
Sep 16 12:38:54.715629 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:54.715555    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerDied","Data":"7852f05f6727101612d7a30a9066c4e5d30a6fae48c6534765912ca7639705db"}
Sep 16 12:38:55.722036 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:55.721817    2559 generic.go:356] "Generic (PLEG): container finished" podID="0a7cb4e6-293e-433b-962e-6b2f052e9732" containerID="68704e9dcb7828161305163e4a61f513840a8fc206dcb40dd62dcb5bb4c85da0" exitCode=0
Sep 16 12:38:55.722036 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:55.721893    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerDied","Data":"68704e9dcb7828161305163e4a61f513840a8fc206dcb40dd62dcb5bb4c85da0"}
Sep 16 12:38:56.033168 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:56.033058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:38:56.033312 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:56.033247    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:38:56.033388 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:38:56.033347    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:00.033305452 +0000 UTC m=+42.116748459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:38:56.728216 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:56.728124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-b69wk" event={"ID":"0a7cb4e6-293e-433b-962e-6b2f052e9732","Type":"ContainerStarted","Data":"aeb3a6c8ba24f520a589e80415f405ac2df101257eddfcddf5a611e06e9dbb70"}
Sep 16 12:38:56.750274 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:56.750220    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-b69wk" podStartSLOduration=8.241014917 podStartE2EDuration="37.750198222s" totalImagesPullingTime="29.509183305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:56.749213749 +0000 UTC m=+38.832656779" watchObservedRunningTime="2026-09-16 12:38:56.750198222 +0000 UTC m=+38.833641237"
Sep 16 12:38:57.732242 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:57.732203    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zsmv" event={"ID":"1aca5a17-6f82-43f7-9b53-c80428d48ac2","Type":"ContainerStarted","Data":"ffcc70085567513b54a4855128f82745e7f7710690d998b748808c0bff48df49"}
Sep 16 12:38:57.750506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:38:57.750457    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zsmv" podStartSLOduration=3.297474636 podStartE2EDuration="5.750435415s" totalImagesPullingTime="2.452960779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:38:57.749859117 +0000 UTC m=+39.833302133" watchObservedRunningTime="2026-09-16 12:38:57.750435415 +0000 UTC m=+39.833878428"
Sep 16 12:39:00.062435 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:00.062379    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:39:00.062812 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:00.062531    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:00.062812 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:00.062595    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:08.062578577 +0000 UTC m=+50.146021568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:39:08.122394 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:08.122355    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:39:08.122938 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:08.122486    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:08.122938 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:08.122558    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:24.122538207 +0000 UTC m=+66.205981213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:39:12.335009 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.334974    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"]
Sep 16 12:39:12.368050 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.368017    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"]
Sep 16 12:39:12.368187 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.368120    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.370251 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.370226    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:39:12.370251 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.370241    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:39:12.370934 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.370914    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:39:12.371649 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.371629    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 12:39:12.554269 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.554225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9144868b-ff17-4099-8398-09226ac7d709-klusterlet-config\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.554457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.554350    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9144868b-ff17-4099-8398-09226ac7d709-tmp\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.554457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.554386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr62s\" (UniqueName: \"kubernetes.io/projected/9144868b-ff17-4099-8398-09226ac7d709-kube-api-access-kr62s\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.654973 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.654902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9144868b-ff17-4099-8398-09226ac7d709-tmp\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.654973 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.654946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr62s\" (UniqueName: \"kubernetes.io/projected/9144868b-ff17-4099-8398-09226ac7d709-kube-api-access-kr62s\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.655114 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.654983    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9144868b-ff17-4099-8398-09226ac7d709-klusterlet-config\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.655290 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.655270    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9144868b-ff17-4099-8398-09226ac7d709-tmp\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.658674 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.658654    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9144868b-ff17-4099-8398-09226ac7d709-klusterlet-config\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.664279 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.664255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr62s\" (UniqueName: \"kubernetes.io/projected/9144868b-ff17-4099-8398-09226ac7d709-kube-api-access-kr62s\") pod \"klusterlet-addon-workmgr-d5f4c4757-hsprk\" (UID: \"9144868b-ff17-4099-8398-09226ac7d709\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.677131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.677102    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:12.959933 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:12.959902    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"]
Sep 16 12:39:12.963449 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:39:12.963424    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9144868b_ff17_4099_8398_09226ac7d709.slice/crio-a0e56dcfe03f16f5c825fcf22425704815d3cc3adc42a213590333fb7e033392 WatchSource:0}: Error finding container a0e56dcfe03f16f5c825fcf22425704815d3cc3adc42a213590333fb7e033392: Status 404 returned error can't find the container with id a0e56dcfe03f16f5c825fcf22425704815d3cc3adc42a213590333fb7e033392
Sep 16 12:39:13.771990 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:13.771945    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk" event={"ID":"9144868b-ff17-4099-8398-09226ac7d709","Type":"ContainerStarted","Data":"a0e56dcfe03f16f5c825fcf22425704815d3cc3adc42a213590333fb7e033392"}
Sep 16 12:39:17.708853 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:17.708828    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j5sd5"
Sep 16 12:39:17.783102 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:17.783062    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk" event={"ID":"9144868b-ff17-4099-8398-09226ac7d709","Type":"ContainerStarted","Data":"ce81f4a78241d368c7693046626bafbae6ab382dd9b09ff352be08ee57f1ad47"}
Sep 16 12:39:17.783310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:17.783291    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:17.784765 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:17.784747    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:39:17.801602 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:17.801548    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk" podStartSLOduration=1.638724897 podStartE2EDuration="5.801533871s" totalImagesPullingTime="4.162808974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:39:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:17.800669022 +0000 UTC m=+59.884112038" watchObservedRunningTime="2026-09-16 12:39:17.801533871 +0000 UTC m=+59.884976885"
Sep 16 12:39:24.127010 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.126974    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:39:24.127010 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.127021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:39:24.127592 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.127060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:39:24.127592 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:24.127145    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:24.127592 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:24.127213    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:39:56.127198182 +0000 UTC m=+98.210641174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:39:24.129553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.129534    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:39:24.129639 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.129601    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:39:24.138071 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:24.138051    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:39:24.138162 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:24.138092    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:40:28.138080611 +0000 UTC m=+130.221523619 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : secret "metrics-daemon-secret" not found
Sep 16 12:39:24.140344 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.140307    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a0f74e27-80ff-462f-a926-9e533448687f-original-pull-secret\") pod \"global-pull-secret-syncer-wn7dw\" (UID: \"a0f74e27-80ff-462f-a926-9e533448687f\") " pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:39:24.328814 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.328764    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:39:24.331200 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.331177    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:39:24.331440 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.331425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wn7dw"
Sep 16 12:39:24.341987 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.341965    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:39:24.353361 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.353313    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28848\" (UniqueName: \"kubernetes.io/projected/c9ec9a45-d9df-439f-b42e-ff9a0f289684-kube-api-access-28848\") pod \"network-check-target-ckvgj\" (UID: \"c9ec9a45-d9df-439f-b42e-ff9a0f289684\") " pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:39:24.598006 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.597963    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wn7dw"]
Sep 16 12:39:24.619962 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.619941    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x7vdr\""
Sep 16 12:39:24.628135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.628114    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:39:24.800728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.800688    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wn7dw" event={"ID":"a0f74e27-80ff-462f-a926-9e533448687f","Type":"ContainerStarted","Data":"0d7fc01c40aa23ed92caf768e790b190274cec5b1bdbc7136a7a109e60f87cd8"}
Sep 16 12:39:24.895247 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:24.895176    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ckvgj"]
Sep 16 12:39:24.899431 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:39:24.899403    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ec9a45_d9df_439f_b42e_ff9a0f289684.slice/crio-ee59b6e0a3dc8ea6487e7fa54ed77b2e3683d4ce8bf46febb67e564e64b46cd6 WatchSource:0}: Error finding container ee59b6e0a3dc8ea6487e7fa54ed77b2e3683d4ce8bf46febb67e564e64b46cd6: Status 404 returned error can't find the container with id ee59b6e0a3dc8ea6487e7fa54ed77b2e3683d4ce8bf46febb67e564e64b46cd6
Sep 16 12:39:25.805112 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:25.805074    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ckvgj" event={"ID":"c9ec9a45-d9df-439f-b42e-ff9a0f289684","Type":"ContainerStarted","Data":"ee59b6e0a3dc8ea6487e7fa54ed77b2e3683d4ce8bf46febb67e564e64b46cd6"}
Sep 16 12:39:29.817809 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:29.817768    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ckvgj" event={"ID":"c9ec9a45-d9df-439f-b42e-ff9a0f289684","Type":"ContainerStarted","Data":"8f772d727e42f188883245bb4d198c824f4c00969a015332dafacde1bea5d994"}
Sep 16 12:39:29.818349 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:29.817867    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:39:29.819373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:29.819353    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wn7dw" event={"ID":"a0f74e27-80ff-462f-a926-9e533448687f","Type":"ContainerStarted","Data":"51d85dd96c770f72f81bab7ec84e846cdc46faca81a7fd5f7318e058e57e8b64"}
Sep 16 12:39:29.835674 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:29.835632    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ckvgj" podStartSLOduration=66.337684222 podStartE2EDuration="1m10.835619127s" totalImagesPullingTime="4.497934905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:29.835252575 +0000 UTC m=+71.918695600" watchObservedRunningTime="2026-09-16 12:39:29.835619127 +0000 UTC m=+71.919062142"
Sep 16 12:39:29.855652 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:29.855613    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wn7dw" podStartSLOduration=66.063456537 podStartE2EDuration="1m10.855599109s" totalImagesPullingTime="4.792142572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:29.854745889 +0000 UTC m=+71.938188905" watchObservedRunningTime="2026-09-16 12:39:29.855599109 +0000 UTC m=+71.939042122"
Sep 16 12:39:56.150780 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:39:56.150652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:39:56.150780 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:56.150770    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:39:56.151296 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:39:56.150848    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls podName:60881a7a-95f1-4713-bddd-7e0e36b2e403 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:00.150828997 +0000 UTC m=+162.234271992 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls") pod "dns-default-vls4q" (UID: "60881a7a-95f1-4713-bddd-7e0e36b2e403") : secret "dns-default-metrics-tls" not found
Sep 16 12:40:00.824340 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:00.824293    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ckvgj"
Sep 16 12:40:28.174015 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:28.173970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:40:28.174531 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:28.174102    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:40:28.174531 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:28.174168    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs podName:9eca049f-d745-4785-b4a2-a85f698c8b3c nodeName:}" failed. No retries permitted until 2026-09-16 12:42:30.174152574 +0000 UTC m=+252.257595566 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs") pod "network-metrics-daemon-nc67r" (UID: "9eca049f-d745-4785-b4a2-a85f698c8b3c") : secret "metrics-daemon-secret" not found
Sep 16 12:40:36.177342 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.177295    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-jtt5w"]
Sep 16 12:40:36.180246 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.180224    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.183365 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.183346    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.183494 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.183478    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jk22m\""
Sep 16 12:40:36.183559 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.183529    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 12:40:36.184260 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.184244    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 12:40:36.194413 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.194389    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.195789 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.195766    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 12:40:36.197100 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.197080    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-jtt5w"]
Sep 16 12:40:36.227502 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.227477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf89e64-8aff-4bdd-a473-0e42517666ed-tmp\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.227586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.227508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0cf89e64-8aff-4bdd-a473-0e42517666ed-snapshots\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.227586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.227532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf89e64-8aff-4bdd-a473-0e42517666ed-serving-cert\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.227653 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.227601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf89e64-8aff-4bdd-a473-0e42517666ed-service-ca-bundle\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.227728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.227654    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf89e64-8aff-4bdd-a473-0e42517666ed-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.227728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.227687    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zghc\" (UniqueName: \"kubernetes.io/projected/0cf89e64-8aff-4bdd-a473-0e42517666ed-kube-api-access-6zghc\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.272617 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.272596    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"]
Sep 16 12:40:36.275214 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.275198    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.276619 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.276600    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"]
Sep 16 12:40:36.277739 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.277717    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 12:40:36.277898 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.277883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 12:40:36.277957 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.277918    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-8ktbg\""
Sep 16 12:40:36.277957 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.277942    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.278187 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.278171    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.279188 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.279171    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b844f9c45-d6jnb"]
Sep 16 12:40:36.279310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.279283    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.281501 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.281480    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 12:40:36.281813 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.281797    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.284204 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.284188    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 12:40:36.284382 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.284366    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 12:40:36.284436 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.284405    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 12:40:36.284643 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.284626    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 12:40:36.284849 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.284829    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lncp2\""
Sep 16 12:40:36.284923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.284831    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:40:36.285288 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.285274    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5ps7f\""
Sep 16 12:40:36.291091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.291068    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 12:40:36.293153 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.293127    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"]
Sep 16 12:40:36.297315 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.297296    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"]
Sep 16 12:40:36.298415 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.298391    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b844f9c45-d6jnb"]
Sep 16 12:40:36.328371 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0cf89e64-8aff-4bdd-a473-0e42517666ed-snapshots\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.328481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf89e64-8aff-4bdd-a473-0e42517666ed-serving-cert\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.328481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328398    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8bz\" (UniqueName: \"kubernetes.io/projected/f648151d-c582-4e77-95ee-77fef2cf536b-kube-api-access-rz8bz\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.328481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328415    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f648151d-c582-4e77-95ee-77fef2cf536b-serving-cert\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.328481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328432    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.328481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/21abcdb6-4e8c-4684-87df-833788d4794e-kube-api-access-s6c26\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.328664 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f648151d-c582-4e77-95ee-77fef2cf536b-config\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.328664 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328514    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf89e64-8aff-4bdd-a473-0e42517666ed-service-ca-bundle\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.328664 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328530    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf89e64-8aff-4bdd-a473-0e42517666ed-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.328664 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328561    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zghc\" (UniqueName: \"kubernetes.io/projected/0cf89e64-8aff-4bdd-a473-0e42517666ed-kube-api-access-6zghc\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.328664 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.328614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf89e64-8aff-4bdd-a473-0e42517666ed-tmp\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.329040 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.329018    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf89e64-8aff-4bdd-a473-0e42517666ed-tmp\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.329161 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.329065    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/0cf89e64-8aff-4bdd-a473-0e42517666ed-snapshots\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.329161 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.329104    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf89e64-8aff-4bdd-a473-0e42517666ed-service-ca-bundle\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.329462 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.329444    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf89e64-8aff-4bdd-a473-0e42517666ed-trusted-ca-bundle\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.330984 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.330965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf89e64-8aff-4bdd-a473-0e42517666ed-serving-cert\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.341991 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.341965    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zghc\" (UniqueName: \"kubernetes.io/projected/0cf89e64-8aff-4bdd-a473-0e42517666ed-kube-api-access-6zghc\") pod \"insights-operator-5844b856db-jtt5w\" (UID: \"0cf89e64-8aff-4bdd-a473-0e42517666ed\") " pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.429174 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429112    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-certificates\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429174 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429146    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429174 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429160    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d12ada3f-c5e3-41ed-a936-680ff68e6e13-ca-trust-extracted\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8bz\" (UniqueName: \"kubernetes.io/projected/f648151d-c582-4e77-95ee-77fef2cf536b-kube-api-access-rz8bz\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.429438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429194    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-trusted-ca\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f648151d-c582-4e77-95ee-77fef2cf536b-serving-cert\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.429438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbld\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-kube-api-access-gzbld\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429343    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.429438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/21abcdb6-4e8c-4684-87df-833788d4794e-kube-api-access-s6c26\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.429694 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429463    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-installation-pull-secrets\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429694 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429495    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f648151d-c582-4e77-95ee-77fef2cf536b-config\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.429694 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.429501    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:40:36.429694 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429549    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-image-registry-private-configuration\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429694 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.429578    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls podName:21abcdb6-4e8c-4684-87df-833788d4794e nodeName:}" failed. No retries permitted until 2026-09-16 12:40:36.929556622 +0000 UTC m=+139.012999634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-86s7m" (UID: "21abcdb6-4e8c-4684-87df-833788d4794e") : secret "samples-operator-tls" not found
Sep 16 12:40:36.429694 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429623    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-bound-sa-token\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.429975 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.429956    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f648151d-c582-4e77-95ee-77fef2cf536b-config\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.431711 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.431694    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f648151d-c582-4e77-95ee-77fef2cf536b-serving-cert\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.437724 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.437695    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8bz\" (UniqueName: \"kubernetes.io/projected/f648151d-c582-4e77-95ee-77fef2cf536b-kube-api-access-rz8bz\") pod \"service-ca-operator-cf7c776b7-dhktq\" (UID: \"f648151d-c582-4e77-95ee-77fef2cf536b\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.437813 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.437772    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6c26\" (UniqueName: \"kubernetes.io/projected/21abcdb6-4e8c-4684-87df-833788d4794e-kube-api-access-s6c26\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.488378 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.488359    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-jtt5w"
Sep 16 12:40:36.530588 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530556    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-image-registry-private-configuration\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530588 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530590    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-bound-sa-token\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-certificates\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d12ada3f-c5e3-41ed-a936-680ff68e6e13-ca-trust-extracted\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-trusted-ca\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbld\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-kube-api-access-gzbld\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.530791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.530768    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-installation-pull-secrets\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.531178 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.530767    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:40:36.531178 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.530828    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b844f9c45-d6jnb: secret "image-registry-tls" not found
Sep 16 12:40:36.531178 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.530905    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls podName:d12ada3f-c5e3-41ed-a936-680ff68e6e13 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:37.030885432 +0000 UTC m=+139.114328447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls") pod "image-registry-5b844f9c45-d6jnb" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13") : secret "image-registry-tls" not found
Sep 16 12:40:36.531178 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.531142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d12ada3f-c5e3-41ed-a936-680ff68e6e13-ca-trust-extracted\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.531696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.531648    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-certificates\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.532457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.532070    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-trusted-ca\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.534260 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.534205    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-installation-pull-secrets\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.534402 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.534359    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-image-registry-private-configuration\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.541086 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.541063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbld\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-kube-api-access-gzbld\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.541807 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.541786    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-bound-sa-token\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:36.585886 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.585845    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"
Sep 16 12:40:36.755451 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.755373    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-jtt5w"]
Sep 16 12:40:36.758464 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:40:36.758438    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf89e64_8aff_4bdd_a473_0e42517666ed.slice/crio-b56c7e10e6d4bd2e5b34b37ac7e7e9095d02bf18bfadb5277a62392cf624a865 WatchSource:0}: Error finding container b56c7e10e6d4bd2e5b34b37ac7e7e9095d02bf18bfadb5277a62392cf624a865: Status 404 returned error can't find the container with id b56c7e10e6d4bd2e5b34b37ac7e7e9095d02bf18bfadb5277a62392cf624a865
Sep 16 12:40:36.856124 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.856094    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq"]
Sep 16 12:40:36.858890 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:40:36.858865    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf648151d_c582_4e77_95ee_77fef2cf536b.slice/crio-a6ee00128921a8346abd77ae1077a8640f30e4e030cfaae074366b44acb58594 WatchSource:0}: Error finding container a6ee00128921a8346abd77ae1077a8640f30e4e030cfaae074366b44acb58594: Status 404 returned error can't find the container with id a6ee00128921a8346abd77ae1077a8640f30e4e030cfaae074366b44acb58594
Sep 16 12:40:36.933778 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.933746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:36.933914 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.933894    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:40:36.933963 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:36.933956    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls podName:21abcdb6-4e8c-4684-87df-833788d4794e nodeName:}" failed. No retries permitted until 2026-09-16 12:40:37.933941507 +0000 UTC m=+140.017384499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-86s7m" (UID: "21abcdb6-4e8c-4684-87df-833788d4794e") : secret "samples-operator-tls" not found
Sep 16 12:40:36.981212 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.981182    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jtt5w" event={"ID":"0cf89e64-8aff-4bdd-a473-0e42517666ed","Type":"ContainerStarted","Data":"b56c7e10e6d4bd2e5b34b37ac7e7e9095d02bf18bfadb5277a62392cf624a865"}
Sep 16 12:40:36.982246 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:36.982226    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq" event={"ID":"f648151d-c582-4e77-95ee-77fef2cf536b","Type":"ContainerStarted","Data":"a6ee00128921a8346abd77ae1077a8640f30e4e030cfaae074366b44acb58594"}
Sep 16 12:40:37.034786 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:37.034733    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:37.034869 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:37.034857    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:40:37.034915 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:37.034871    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b844f9c45-d6jnb: secret "image-registry-tls" not found
Sep 16 12:40:37.034950 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:37.034917    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls podName:d12ada3f-c5e3-41ed-a936-680ff68e6e13 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:38.03490069 +0000 UTC m=+140.118343683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls") pod "image-registry-5b844f9c45-d6jnb" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13") : secret "image-registry-tls" not found
Sep 16 12:40:37.944747 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:37.944705    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:37.945200 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:37.944895    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:40:37.945200 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:37.944981    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls podName:21abcdb6-4e8c-4684-87df-833788d4794e nodeName:}" failed. No retries permitted until 2026-09-16 12:40:39.944959907 +0000 UTC m=+142.028402903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-86s7m" (UID: "21abcdb6-4e8c-4684-87df-833788d4794e") : secret "samples-operator-tls" not found
Sep 16 12:40:38.045382 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.045340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:38.045566 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:38.045484    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:40:38.045566 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:38.045500    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b844f9c45-d6jnb: secret "image-registry-tls" not found
Sep 16 12:40:38.045566 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:38.045564    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls podName:d12ada3f-c5e3-41ed-a936-680ff68e6e13 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:40.045546339 +0000 UTC m=+142.128989337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls") pod "image-registry-5b844f9c45-d6jnb" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13") : secret "image-registry-tls" not found
Sep 16 12:40:38.675384 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.675351    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"]
Sep 16 12:40:38.678368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.678344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:38.680933 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.680910    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 12:40:38.681044 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.680958    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-785jg\""
Sep 16 12:40:38.681044 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.680960    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 12:40:38.688375 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.688354    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"]
Sep 16 12:40:38.750618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.750587    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85f9f824-9735-4103-bae9-8ba69dd5fae8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:38.750797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.750656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:38.851842 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.851804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85f9f824-9735-4103-bae9-8ba69dd5fae8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:38.852016 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.851876    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:38.852016 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:38.852003    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:40:38.852092 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:38.852072    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert podName:85f9f824-9735-4103-bae9-8ba69dd5fae8 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:39.352052694 +0000 UTC m=+141.435495686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2xbhj" (UID: "85f9f824-9735-4103-bae9-8ba69dd5fae8") : secret "networking-console-plugin-cert" not found
Sep 16 12:40:38.852508 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:38.852488    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/85f9f824-9735-4103-bae9-8ba69dd5fae8-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:39.357431 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:39.357380    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:39.357896 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:39.357578    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:40:39.357896 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:39.357665    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert podName:85f9f824-9735-4103-bae9-8ba69dd5fae8 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:40.357644004 +0000 UTC m=+142.441087014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2xbhj" (UID: "85f9f824-9735-4103-bae9-8ba69dd5fae8") : secret "networking-console-plugin-cert" not found
Sep 16 12:40:39.962302 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:39.962259    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:39.962508 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:39.962454    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:40:39.962573 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:39.962540    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls podName:21abcdb6-4e8c-4684-87df-833788d4794e nodeName:}" failed. No retries permitted until 2026-09-16 12:40:43.962517084 +0000 UTC m=+146.045960100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-86s7m" (UID: "21abcdb6-4e8c-4684-87df-833788d4794e") : secret "samples-operator-tls" not found
Sep 16 12:40:39.994003 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:39.993969    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jtt5w" event={"ID":"0cf89e64-8aff-4bdd-a473-0e42517666ed","Type":"ContainerStarted","Data":"8a18565645b5ba13198f7f3e5858c5806bab042ffbcfd673abd04ac5ccf871f5"}
Sep 16 12:40:39.995448 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:39.995415    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq" event={"ID":"f648151d-c582-4e77-95ee-77fef2cf536b","Type":"ContainerStarted","Data":"36fe766eebe52b343ab25eea0412ff58da1418514ff671909620ec5d45f25ef5"}
Sep 16 12:40:40.015573 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.015527    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-jtt5w" podStartSLOduration=1.2838160250000001 podStartE2EDuration="4.015513852s" totalImagesPullingTime="2.731697827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:40.014897058 +0000 UTC m=+142.098340099" watchObservedRunningTime="2026-09-16 12:40:40.015513852 +0000 UTC m=+142.098956866"
Sep 16 12:40:40.032433 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.032382    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq" podStartSLOduration=1.397502858 podStartE2EDuration="4.032365505s" totalImagesPullingTime="2.634862647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:40.031394934 +0000 UTC m=+142.114837949" watchObservedRunningTime="2026-09-16 12:40:40.032365505 +0000 UTC m=+142.115808523"
Sep 16 12:40:40.063926 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.063346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:40.063926 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:40.063492    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:40:40.063926 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:40.063509    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b844f9c45-d6jnb: secret "image-registry-tls" not found
Sep 16 12:40:40.063926 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:40.063565    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls podName:d12ada3f-c5e3-41ed-a936-680ff68e6e13 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.063545927 +0000 UTC m=+146.146988921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls") pod "image-registry-5b844f9c45-d6jnb" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13") : secret "image-registry-tls" not found
Sep 16 12:40:40.365598 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.365503    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:40.365956 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:40.365637    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:40:40.365956 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:40.365700    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert podName:85f9f824-9735-4103-bae9-8ba69dd5fae8 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:42.365685979 +0000 UTC m=+144.449128984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2xbhj" (UID: "85f9f824-9735-4103-bae9-8ba69dd5fae8") : secret "networking-console-plugin-cert" not found
Sep 16 12:40:40.734708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.734677    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"]
Sep 16 12:40:40.737981 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.737960    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"
Sep 16 12:40:40.740215 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.740190    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 12:40:40.740501 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.740485    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8nr9x\""
Sep 16 12:40:40.741028 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.741012    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 12:40:40.749877 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.749859    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"]
Sep 16 12:40:40.870138 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.870102    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rwv\" (UniqueName: \"kubernetes.io/projected/00183af1-d5ba-4233-ab73-f8f123d6e09e-kube-api-access-62rwv\") pod \"migrator-8b4d697c5-mswrn\" (UID: \"00183af1-d5ba-4233-ab73-f8f123d6e09e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"
Sep 16 12:40:40.970477 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.970444    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rwv\" (UniqueName: \"kubernetes.io/projected/00183af1-d5ba-4233-ab73-f8f123d6e09e-kube-api-access-62rwv\") pod \"migrator-8b4d697c5-mswrn\" (UID: \"00183af1-d5ba-4233-ab73-f8f123d6e09e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"
Sep 16 12:40:40.978730 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:40.978703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rwv\" (UniqueName: \"kubernetes.io/projected/00183af1-d5ba-4233-ab73-f8f123d6e09e-kube-api-access-62rwv\") pod \"migrator-8b4d697c5-mswrn\" (UID: \"00183af1-d5ba-4233-ab73-f8f123d6e09e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"
Sep 16 12:40:41.047350 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.047257    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"
Sep 16 12:40:41.133581 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.133554    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8lq9"]
Sep 16 12:40:41.136854 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.136838    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.139708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.139690    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9z4xc\""
Sep 16 12:40:41.139800 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.139712    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:40:41.139916 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.139898    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:40:41.150574 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.150554    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8lq9"]
Sep 16 12:40:41.273216 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.273183    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.273216 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.273217    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f883c3a1-fb07-4fbb-998a-4fea855834c0-crio-socket\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.273457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.273261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.273457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.273361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f883c3a1-fb07-4fbb-998a-4fea855834c0-data-volume\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.273457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.273418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwl8r\" (UniqueName: \"kubernetes.io/projected/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-api-access-zwl8r\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.326943 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.326869    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn"]
Sep 16 12:40:41.330255 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:40:41.330228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00183af1_d5ba_4233_ab73_f8f123d6e09e.slice/crio-69ff2e7e254994faac646d854cb78610ffd324dcb53319b57a33a2d4dc1c92d2 WatchSource:0}: Error finding container 69ff2e7e254994faac646d854cb78610ffd324dcb53319b57a33a2d4dc1c92d2: Status 404 returned error can't find the container with id 69ff2e7e254994faac646d854cb78610ffd324dcb53319b57a33a2d4dc1c92d2
Sep 16 12:40:41.374297 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwl8r\" (UniqueName: \"kubernetes.io/projected/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-api-access-zwl8r\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374346    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374375    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f883c3a1-fb07-4fbb-998a-4fea855834c0-crio-socket\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.374428    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374468    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f883c3a1-fb07-4fbb-998a-4fea855834c0-data-volume\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.374490    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:41.874475746 +0000 UTC m=+143.957918764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374503    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f883c3a1-fb07-4fbb-998a-4fea855834c0-crio-socket\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.374597    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:41.374641 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.374625    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:41.874617521 +0000 UTC m=+143.958060541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:41.374918 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.374745    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f883c3a1-fb07-4fbb-998a-4fea855834c0-data-volume\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.385181 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.385153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwl8r\" (UniqueName: \"kubernetes.io/projected/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-api-access-zwl8r\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.879935 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.879901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.880185 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:41.879964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:41.880185 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.880034    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.880185 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.880091    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:41.880966 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.880266    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:42.880086997 +0000 UTC m=+144.963530006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:41.880966 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:41.880532    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:42.880509239 +0000 UTC m=+144.963952243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:42.005359 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:42.005298    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn" event={"ID":"00183af1-d5ba-4233-ab73-f8f123d6e09e","Type":"ContainerStarted","Data":"69ff2e7e254994faac646d854cb78610ffd324dcb53319b57a33a2d4dc1c92d2"}
Sep 16 12:40:42.384061 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:42.384024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:42.384529 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:42.384175    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:40:42.384529 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:42.384236    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert podName:85f9f824-9735-4103-bae9-8ba69dd5fae8 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:46.38422138 +0000 UTC m=+148.467664390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2xbhj" (UID: "85f9f824-9735-4103-bae9-8ba69dd5fae8") : secret "networking-console-plugin-cert" not found
Sep 16 12:40:42.889513 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:42.889422    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:42.889513 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:42.889477    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:42.889729 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:42.889561    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:42.889729 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:42.889574    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:42.889729 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:42.889642    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.889622438 +0000 UTC m=+146.973065448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:42.889729 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:42.889663    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:44.889652806 +0000 UTC m=+146.973095814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:43.010260 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:43.010223    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn" event={"ID":"00183af1-d5ba-4233-ab73-f8f123d6e09e","Type":"ContainerStarted","Data":"750430a9b92d9b32e0e7b1f5623d98c41c5154c6c243ab93492e8e1714569cb4"}
Sep 16 12:40:43.010433 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:43.010281    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn" event={"ID":"00183af1-d5ba-4233-ab73-f8f123d6e09e","Type":"ContainerStarted","Data":"576a72566b09be23525b736621cb3855eb6abdb358255b0eb5a36dee2724652f"}
Sep 16 12:40:43.040065 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:43.040015    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-mswrn" podStartSLOduration=1.736420543 podStartE2EDuration="3.040000408s" totalImagesPullingTime="1.303579865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:43.039831323 +0000 UTC m=+145.123274354" watchObservedRunningTime="2026-09-16 12:40:43.040000408 +0000 UTC m=+145.123443460"
Sep 16 12:40:43.998797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:43.998763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:43.999186 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:43.998884    2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:40:43.999186 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:43.998934    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls podName:21abcdb6-4e8c-4684-87df-833788d4794e nodeName:}" failed. No retries permitted until 2026-09-16 12:40:51.998921759 +0000 UTC m=+154.082364751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-86s7m" (UID: "21abcdb6-4e8c-4684-87df-833788d4794e") : secret "samples-operator-tls" not found
Sep 16 12:40:44.099933 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:44.099895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:44.100111 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.100002    2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 12:40:44.100111 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.100014    2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b844f9c45-d6jnb: secret "image-registry-tls" not found
Sep 16 12:40:44.100111 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.100062    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls podName:d12ada3f-c5e3-41ed-a936-680ff68e6e13 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:52.100049176 +0000 UTC m=+154.183492168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls") pod "image-registry-5b844f9c45-d6jnb" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13") : secret "image-registry-tls" not found
Sep 16 12:40:44.187236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:44.187211    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v47l6_9fb924aa-f0e1-46e9-97d6-e5db3cdd500c/dns-node-resolver/0.log"
Sep 16 12:40:44.905369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:44.905314    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:44.905542 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:44.905401    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:44.905542 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.905439    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:44.905542 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.905504    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:48.905489509 +0000 UTC m=+150.988932520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:44.905542 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.905540    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:44.905674 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:44.905589    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:48.905577817 +0000 UTC m=+150.989020822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:45.186074 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:45.186047    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxn2l_c48de37d-b52d-44e7-bd6a-4515f7011d47/node-ca/0.log"
Sep 16 12:40:45.586618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:45.586537    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zsmv_1aca5a17-6f82-43f7-9b53-c80428d48ac2/serve-healthcheck-canary/0.log"
Sep 16 12:40:46.187171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:46.187138    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mswrn_00183af1-d5ba-4233-ab73-f8f123d6e09e/migrator/0.log"
Sep 16 12:40:46.385523 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:46.385495    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-mswrn_00183af1-d5ba-4233-ab73-f8f123d6e09e/graceful-termination/0.log"
Sep 16 12:40:46.419061 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:46.419039    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:46.419201 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:46.419131    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:40:46.419201 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:46.419181    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert podName:85f9f824-9735-4103-bae9-8ba69dd5fae8 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:54.419167883 +0000 UTC m=+156.502610875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2xbhj" (UID: "85f9f824-9735-4103-bae9-8ba69dd5fae8") : secret "networking-console-plugin-cert" not found
Sep 16 12:40:48.939930 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:48.939887    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:48.940374 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:48.939949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:48.940374 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:48.940022    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:40:48.940374 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:48.940065    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:48.940374 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:48.940096    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:56.940080817 +0000 UTC m=+159.023523808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : configmap "kube-rbac-proxy" not found
Sep 16 12:40:48.940374 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:48.940117    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:40:56.940103384 +0000 UTC m=+159.023546392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:52.066785 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.066731    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:52.069255 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.069233    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/21abcdb6-4e8c-4684-87df-833788d4794e-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-86s7m\" (UID: \"21abcdb6-4e8c-4684-87df-833788d4794e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:52.168273 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.168232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:52.170859 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.170836    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"image-registry-5b844f9c45-d6jnb\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") " pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:52.191906 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.191873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"
Sep 16 12:40:52.201116 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.201090    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:52.466622 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.466585    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m"]
Sep 16 12:40:52.492175 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:52.492143    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b844f9c45-d6jnb"]
Sep 16 12:40:52.495454 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:40:52.495426    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12ada3f_c5e3_41ed_a936_680ff68e6e13.slice/crio-606ae1e9b9133d8dcceccc463dfb451021e3922b7d5111478521218fc2076c67 WatchSource:0}: Error finding container 606ae1e9b9133d8dcceccc463dfb451021e3922b7d5111478521218fc2076c67: Status 404 returned error can't find the container with id 606ae1e9b9133d8dcceccc463dfb451021e3922b7d5111478521218fc2076c67
Sep 16 12:40:53.042356 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:53.042299    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m" event={"ID":"21abcdb6-4e8c-4684-87df-833788d4794e","Type":"ContainerStarted","Data":"f22f2aa7da5b2bda1f756ddd76fd81e5d0f2e159d9cbf999a50d13f17a6a19e0"}
Sep 16 12:40:53.044023 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:53.043994    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb" event={"ID":"d12ada3f-c5e3-41ed-a936-680ff68e6e13","Type":"ContainerStarted","Data":"6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18"}
Sep 16 12:40:53.044157 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:53.044044    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb" event={"ID":"d12ada3f-c5e3-41ed-a936-680ff68e6e13","Type":"ContainerStarted","Data":"606ae1e9b9133d8dcceccc463dfb451021e3922b7d5111478521218fc2076c67"}
Sep 16 12:40:53.044157 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:53.044143    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:40:53.065865 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:53.065803    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb" podStartSLOduration=17.065782157 podStartE2EDuration="17.065782157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:53.063579116 +0000 UTC m=+155.147022155" watchObservedRunningTime="2026-09-16 12:40:53.065782157 +0000 UTC m=+155.149225204"
Sep 16 12:40:54.488713 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:54.488676    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:40:54.489081 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:54.488850    2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 12:40:54.489081 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:54.488938    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert podName:85f9f824-9735-4103-bae9-8ba69dd5fae8 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:10.488920149 +0000 UTC m=+172.572363154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-2xbhj" (UID: "85f9f824-9735-4103-bae9-8ba69dd5fae8") : secret "networking-console-plugin-cert" not found
Sep 16 12:40:55.053782 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:55.053740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m" event={"ID":"21abcdb6-4e8c-4684-87df-833788d4794e","Type":"ContainerStarted","Data":"918ca3292991f6da775cc9c219672855d23ac7037a01bf90479098fbf539d087"}
Sep 16 12:40:55.053936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:55.053806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m" event={"ID":"21abcdb6-4e8c-4684-87df-833788d4794e","Type":"ContainerStarted","Data":"5e288efe3dc7b557ecbe6e7aa872ddb2c8878930792641761ff0335a99145cdc"}
Sep 16 12:40:55.070603 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:55.070555    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-86s7m" podStartSLOduration=17.566690567 podStartE2EDuration="19.070537158s" totalImagesPullingTime="1.503846591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:40:55.069820639 +0000 UTC m=+157.153263656" watchObservedRunningTime="2026-09-16 12:40:55.070537158 +0000 UTC m=+157.153980175"
Sep 16 12:40:55.345695 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:55.345605    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vls4q" podUID="60881a7a-95f1-4713-bddd-7e0e36b2e403"
Sep 16 12:40:56.058641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:56.058606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vls4q"
Sep 16 12:40:56.426377 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:56.426341    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-nc67r" podUID="9eca049f-d745-4785-b4a2-a85f698c8b3c"
Sep 16 12:40:57.011560 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:57.011526    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:57.011753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:57.011586    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:40:57.011753 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:57.011681    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:57.011753 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:40:57.011734    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls podName:f883c3a1-fb07-4fbb-998a-4fea855834c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:41:13.01172156 +0000 UTC m=+175.095164566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8lq9" (UID: "f883c3a1-fb07-4fbb-998a-4fea855834c0") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:40:57.012123 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:40:57.012099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f883c3a1-fb07-4fbb-998a-4fea855834c0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:41:00.236344 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:00.236283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:41:00.238752 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:00.238727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/60881a7a-95f1-4713-bddd-7e0e36b2e403-metrics-tls\") pod \"dns-default-vls4q\" (UID: \"60881a7a-95f1-4713-bddd-7e0e36b2e403\") " pod="openshift-dns/dns-default-vls4q"
Sep 16 12:41:00.261957 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:00.261922    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fgm9v\""
Sep 16 12:41:00.270750 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:00.270725    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vls4q"
Sep 16 12:41:00.541153 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:00.541080    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vls4q"]
Sep 16 12:41:00.544041 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:00.544013    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60881a7a_95f1_4713_bddd_7e0e36b2e403.slice/crio-58af5930d67bcfad9ce43440244b6becb25842734874127f7395ae8dd93af61f WatchSource:0}: Error finding container 58af5930d67bcfad9ce43440244b6becb25842734874127f7395ae8dd93af61f: Status 404 returned error can't find the container with id 58af5930d67bcfad9ce43440244b6becb25842734874127f7395ae8dd93af61f
Sep 16 12:41:01.076115 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:01.076064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vls4q" event={"ID":"60881a7a-95f1-4713-bddd-7e0e36b2e403","Type":"ContainerStarted","Data":"58af5930d67bcfad9ce43440244b6becb25842734874127f7395ae8dd93af61f"}
Sep 16 12:41:02.081544 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:02.081512    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vls4q" event={"ID":"60881a7a-95f1-4713-bddd-7e0e36b2e403","Type":"ContainerStarted","Data":"de346b6881724cde3fde2ae7f174b8d9780bf475f9d0dd39c5e271a3ad1bc4e8"}
Sep 16 12:41:03.086253 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:03.086218    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vls4q" event={"ID":"60881a7a-95f1-4713-bddd-7e0e36b2e403","Type":"ContainerStarted","Data":"246aed06a5dfbbd49b929923711290fd05188471852cfb205679fcd8a8677797"}
Sep 16 12:41:03.086652 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:03.086287    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vls4q"
Sep 16 12:41:03.104762 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:03.104711    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vls4q" podStartSLOduration=129.747376417 podStartE2EDuration="2m11.104693788s" totalImagesPullingTime="1.357317371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:03.103584788 +0000 UTC m=+165.187027814" watchObservedRunningTime="2026-09-16 12:41:03.104693788 +0000 UTC m=+165.188136801"
Sep 16 12:41:04.671448 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.671416    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b844f9c45-d6jnb"]
Sep 16 12:41:04.739029 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.738995    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b9c897d6f-pxckm"]
Sep 16 12:41:04.742073 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.742049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.761406 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.761378    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9c897d6f-pxckm"]
Sep 16 12:41:04.774829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.774801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3aa768c-66e0-4780-8362-08249fc9f1cc-registry-certificates\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.774829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.774832    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3aa768c-66e0-4780-8362-08249fc9f1cc-trusted-ca\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.775002 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.774850    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slbb\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-kube-api-access-6slbb\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.775002 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.774892    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-bound-sa-token\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.775002 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.774979    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3aa768c-66e0-4780-8362-08249fc9f1cc-ca-trust-extracted\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.775098 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.775022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3aa768c-66e0-4780-8362-08249fc9f1cc-image-registry-private-configuration\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.775098 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.775055    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3aa768c-66e0-4780-8362-08249fc9f1cc-installation-pull-secrets\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.775098 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.775084    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-registry-tls\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875520    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-registry-tls\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875743 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3aa768c-66e0-4780-8362-08249fc9f1cc-registry-certificates\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875743 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875589    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3aa768c-66e0-4780-8362-08249fc9f1cc-trusted-ca\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875743 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875606    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slbb\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-kube-api-access-6slbb\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875743 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-bound-sa-token\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875743 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875670    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3aa768c-66e0-4780-8362-08249fc9f1cc-ca-trust-extracted\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.875743 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3aa768c-66e0-4780-8362-08249fc9f1cc-image-registry-private-configuration\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.876036 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.875896    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3aa768c-66e0-4780-8362-08249fc9f1cc-installation-pull-secrets\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.876125 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.876105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3aa768c-66e0-4780-8362-08249fc9f1cc-ca-trust-extracted\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.876462 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.876439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3aa768c-66e0-4780-8362-08249fc9f1cc-registry-certificates\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.876775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.876752    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3aa768c-66e0-4780-8362-08249fc9f1cc-trusted-ca\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.878291 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.878266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3aa768c-66e0-4780-8362-08249fc9f1cc-installation-pull-secrets\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.878402 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.878303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3aa768c-66e0-4780-8362-08249fc9f1cc-image-registry-private-configuration\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.878402 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.878389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-registry-tls\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.887262 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.887229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-bound-sa-token\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:04.887791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:04.887772    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slbb\" (UniqueName: \"kubernetes.io/projected/b3aa768c-66e0-4780-8362-08249fc9f1cc-kube-api-access-6slbb\") pod \"image-registry-6b9c897d6f-pxckm\" (UID: \"b3aa768c-66e0-4780-8362-08249fc9f1cc\") " pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:05.050520 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:05.050432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:05.327969 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:05.327888    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b9c897d6f-pxckm"]
Sep 16 12:41:05.330760 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:05.330732    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aa768c_66e0_4780_8362_08249fc9f1cc.slice/crio-5b95c961c38c1e583fd11db0bc637bf74b370e71dcfc853ef40a3c144fc45e9a WatchSource:0}: Error finding container 5b95c961c38c1e583fd11db0bc637bf74b370e71dcfc853ef40a3c144fc45e9a: Status 404 returned error can't find the container with id 5b95c961c38c1e583fd11db0bc637bf74b370e71dcfc853ef40a3c144fc45e9a
Sep 16 12:41:06.096448 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:06.096408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm" event={"ID":"b3aa768c-66e0-4780-8362-08249fc9f1cc","Type":"ContainerStarted","Data":"06b49cfaaf1d0493742244a521516187c1f6eeffad5e397caf30f0a232dcbdc6"}
Sep 16 12:41:06.096448 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:06.096454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm" event={"ID":"b3aa768c-66e0-4780-8362-08249fc9f1cc","Type":"ContainerStarted","Data":"5b95c961c38c1e583fd11db0bc637bf74b370e71dcfc853ef40a3c144fc45e9a"}
Sep 16 12:41:06.096848 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:06.096502    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:06.119464 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:06.119420    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm" podStartSLOduration=2.119402492 podStartE2EDuration="2.119402492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:06.117276885 +0000 UTC m=+168.200719898" watchObservedRunningTime="2026-09-16 12:41:06.119402492 +0000 UTC m=+168.202845505"
Sep 16 12:41:10.519430 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:10.519340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:41:10.521748 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:10.521729    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/85f9f824-9735-4103-bae9-8ba69dd5fae8-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-2xbhj\" (UID: \"85f9f824-9735-4103-bae9-8ba69dd5fae8\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:41:10.788909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:10.788800    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"
Sep 16 12:41:11.065982 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:11.065911    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj"]
Sep 16 12:41:11.069443 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:11.069419    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f9f824_9735_4103_bae9_8ba69dd5fae8.slice/crio-51169b138379c5e6e905d0671fb72f38a54fb395425933d3faead7bd7713257c WatchSource:0}: Error finding container 51169b138379c5e6e905d0671fb72f38a54fb395425933d3faead7bd7713257c: Status 404 returned error can't find the container with id 51169b138379c5e6e905d0671fb72f38a54fb395425933d3faead7bd7713257c
Sep 16 12:41:11.115752 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:11.115719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj" event={"ID":"85f9f824-9735-4103-bae9-8ba69dd5fae8","Type":"ContainerStarted","Data":"51169b138379c5e6e905d0671fb72f38a54fb395425933d3faead7bd7713257c"}
Sep 16 12:41:11.405800 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:11.405709    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:41:12.120940 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:12.120902    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj" event={"ID":"85f9f824-9735-4103-bae9-8ba69dd5fae8","Type":"ContainerStarted","Data":"c8ea1f56c70724ef65a556dd2c27d683d74e1998f0d00e2b0b0572b327aec685"}
Sep 16 12:41:12.138667 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:12.138622    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-2xbhj" podStartSLOduration=33.17035586 podStartE2EDuration="34.138607113s" totalImagesPullingTime="968.251253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:12.137579989 +0000 UTC m=+174.221023037" watchObservedRunningTime="2026-09-16 12:41:12.138607113 +0000 UTC m=+174.222050126"
Sep 16 12:41:13.040137 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:13.040104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:41:13.042727 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:13.042698    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f883c3a1-fb07-4fbb-998a-4fea855834c0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8lq9\" (UID: \"f883c3a1-fb07-4fbb-998a-4fea855834c0\") " pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:41:13.091101 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:13.091076    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vls4q"
Sep 16 12:41:13.246820 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:13.246791    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8lq9"
Sep 16 12:41:13.520210 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:13.520177    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8lq9"]
Sep 16 12:41:13.523041 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:13.523016    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf883c3a1_fb07_4fbb_998a_4fea855834c0.slice/crio-19fdb44bb25bfc8289994037c76878267ee733053d0108ab5f091a2fb8e7714c WatchSource:0}: Error finding container 19fdb44bb25bfc8289994037c76878267ee733053d0108ab5f091a2fb8e7714c: Status 404 returned error can't find the container with id 19fdb44bb25bfc8289994037c76878267ee733053d0108ab5f091a2fb8e7714c
Sep 16 12:41:14.133906 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:14.133864    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8lq9" event={"ID":"f883c3a1-fb07-4fbb-998a-4fea855834c0","Type":"ContainerStarted","Data":"884a6adb8ce3815bbb6fea90c03688f8e529f5e773e861c89cb51ba30954a937"}
Sep 16 12:41:14.134079 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:14.133935    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8lq9" event={"ID":"f883c3a1-fb07-4fbb-998a-4fea855834c0","Type":"ContainerStarted","Data":"19fdb44bb25bfc8289994037c76878267ee733053d0108ab5f091a2fb8e7714c"}
Sep 16 12:41:14.676528 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:14.676501    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:41:15.139989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:15.139943    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8lq9" event={"ID":"f883c3a1-fb07-4fbb-998a-4fea855834c0","Type":"ContainerStarted","Data":"f633968d4aac2907b18a2a852b95315eed4f34b412585c8f5b9c82b1bcca59df"}
Sep 16 12:41:16.145155 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:16.145108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8lq9" event={"ID":"f883c3a1-fb07-4fbb-998a-4fea855834c0","Type":"ContainerStarted","Data":"2f00e7ededcc1a46111598ef4cbfc3c5230c645270fa0735cd73c61b9182593d"}
Sep 16 12:41:16.167403 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:16.167288    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f8lq9" podStartSLOduration=33.003804453 podStartE2EDuration="35.16727163s" totalImagesPullingTime="2.163467177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:40:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:16.16571813 +0000 UTC m=+178.249161165" watchObservedRunningTime="2026-09-16 12:41:16.16727163 +0000 UTC m=+178.250714645"
Sep 16 12:41:17.783762 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:17.783716    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk" podUID="9144868b-ff17-4099-8398-09226ac7d709" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Sep 16 12:41:18.152596 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:18.152518    2559 generic.go:356] "Generic (PLEG): container finished" podID="9144868b-ff17-4099-8398-09226ac7d709" containerID="ce81f4a78241d368c7693046626bafbae6ab382dd9b09ff352be08ee57f1ad47" exitCode=1
Sep 16 12:41:18.152730 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:18.152594    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk" event={"ID":"9144868b-ff17-4099-8398-09226ac7d709","Type":"ContainerDied","Data":"ce81f4a78241d368c7693046626bafbae6ab382dd9b09ff352be08ee57f1ad47"}
Sep 16 12:41:18.152935 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:18.152921    2559 scope.go:117] "RemoveContainer" containerID="ce81f4a78241d368c7693046626bafbae6ab382dd9b09ff352be08ee57f1ad47"
Sep 16 12:41:19.157099 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:19.157070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk" event={"ID":"9144868b-ff17-4099-8398-09226ac7d709","Type":"ContainerStarted","Data":"804d2119fdc26ee91a9344586e0628e4a12e399ae36fc92803d583786526a1e1"}
Sep 16 12:41:19.157515 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:19.157355    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:41:19.157935 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:19.157919    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d5f4c4757-hsprk"
Sep 16 12:41:27.105256 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:27.105228    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b9c897d6f-pxckm"
Sep 16 12:41:29.169849 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.169814    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b9dw2"]
Sep 16 12:41:29.173967 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.173950    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.176884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.176861    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:41:29.177335 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.177293    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:41:29.177899 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.177883    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:41:29.192844 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.192812    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:41:29.193019 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.192930    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:41:29.193063 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.193033    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:41:29.193096 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.193077    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-krqlf\""
Sep 16 12:41:29.271194 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271156    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwggb\" (UniqueName: \"kubernetes.io/projected/664066a5-a1d8-4be9-ad02-ab9a485e70b0-kube-api-access-fwggb\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271243    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271271    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664066a5-a1d8-4be9-ad02-ab9a485e70b0-metrics-client-ca\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271315    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-root\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-textfile\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271529 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-tls\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271529 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271397    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-wtmp\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.271529 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.271444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-sys\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.372807 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372766    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwggb\" (UniqueName: \"kubernetes.io/projected/664066a5-a1d8-4be9-ad02-ab9a485e70b0-kube-api-access-fwggb\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.372807 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372830    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664066a5-a1d8-4be9-ad02-ab9a485e70b0-metrics-client-ca\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372878    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-root\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-textfile\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-tls\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-wtmp\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372962    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-sys\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372985    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-root\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373038 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.372996    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373444 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.373096    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-sys\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373444 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.373185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-wtmp\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373444 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.373299    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-textfile\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373588 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.373480    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.373645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.373623    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/664066a5-a1d8-4be9-ad02-ab9a485e70b0-metrics-client-ca\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.375488 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.375460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.375589 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.375546    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/664066a5-a1d8-4be9-ad02-ab9a485e70b0-node-exporter-tls\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.381314 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.381279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwggb\" (UniqueName: \"kubernetes.io/projected/664066a5-a1d8-4be9-ad02-ab9a485e70b0-kube-api-access-fwggb\") pod \"node-exporter-b9dw2\" (UID: \"664066a5-a1d8-4be9-ad02-ab9a485e70b0\") " pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.483332 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.483254    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9dw2"
Sep 16 12:41:29.494369 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:29.494308    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod664066a5_a1d8_4be9_ad02_ab9a485e70b0.slice/crio-46c5389e6dd5e31ea1661ee04e9b97c4a710585e48e8d2eae604de9eff0fc0c6 WatchSource:0}: Error finding container 46c5389e6dd5e31ea1661ee04e9b97c4a710585e48e8d2eae604de9eff0fc0c6: Status 404 returned error can't find the container with id 46c5389e6dd5e31ea1661ee04e9b97c4a710585e48e8d2eae604de9eff0fc0c6
Sep 16 12:41:29.690555 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.690506    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb" podUID="d12ada3f-c5e3-41ed-a936-680ff68e6e13" containerName="registry" containerID="cri-o://6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18" gracePeriod=30
Sep 16 12:41:29.922063 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.922038    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:41:29.978120 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978086    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d12ada3f-c5e3-41ed-a936-680ff68e6e13-ca-trust-extracted\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978120 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978133    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzbld\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-kube-api-access-gzbld\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978161    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-image-registry-private-configuration\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-installation-pull-secrets\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978244    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-bound-sa-token\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978265    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-certificates\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978305    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978362    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-trusted-ca\") pod \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\" (UID: \"d12ada3f-c5e3-41ed-a936-680ff68e6e13\") "
Sep 16 12:41:29.978936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978776    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:29.978936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.978874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:29.980866 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.980838    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:29.980960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.980884    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:29.980960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.980939    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-kube-api-access-gzbld" (OuterVolumeSpecName: "kube-api-access-gzbld") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "kube-api-access-gzbld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:29.981045 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.981018    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:29.981153 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.981138    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:29.986616 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:29.986593    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12ada3f-c5e3-41ed-a936-680ff68e6e13-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d12ada3f-c5e3-41ed-a936-680ff68e6e13" (UID: "d12ada3f-c5e3-41ed-a936-680ff68e6e13"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:30.079797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079711    2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-installation-pull-secrets\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.079797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079741    2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-bound-sa-token\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.079797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079754    2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-certificates\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.079797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079767    2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-registry-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.079797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079780    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d12ada3f-c5e3-41ed-a936-680ff68e6e13-trusted-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.079797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079791    2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d12ada3f-c5e3-41ed-a936-680ff68e6e13-ca-trust-extracted\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.080126 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079803    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbld\" (UniqueName: \"kubernetes.io/projected/d12ada3f-c5e3-41ed-a936-680ff68e6e13-kube-api-access-gzbld\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.080126 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.079816    2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d12ada3f-c5e3-41ed-a936-680ff68e6e13-image-registry-private-configuration\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:41:30.197561 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.197523    2559 generic.go:356] "Generic (PLEG): container finished" podID="d12ada3f-c5e3-41ed-a936-680ff68e6e13" containerID="6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18" exitCode=0
Sep 16 12:41:30.197987 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.197575    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb"
Sep 16 12:41:30.197987 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.197575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb" event={"ID":"d12ada3f-c5e3-41ed-a936-680ff68e6e13","Type":"ContainerDied","Data":"6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18"}
Sep 16 12:41:30.197987 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.197641    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b844f9c45-d6jnb" event={"ID":"d12ada3f-c5e3-41ed-a936-680ff68e6e13","Type":"ContainerDied","Data":"606ae1e9b9133d8dcceccc463dfb451021e3922b7d5111478521218fc2076c67"}
Sep 16 12:41:30.198469 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.198448    2559 scope.go:117] "RemoveContainer" containerID="6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18"
Sep 16 12:41:30.199367 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.199337    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9dw2" event={"ID":"664066a5-a1d8-4be9-ad02-ab9a485e70b0","Type":"ContainerStarted","Data":"46c5389e6dd5e31ea1661ee04e9b97c4a710585e48e8d2eae604de9eff0fc0c6"}
Sep 16 12:41:30.208524 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.208507    2559 scope.go:117] "RemoveContainer" containerID="6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18"
Sep 16 12:41:30.208944 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:41:30.208921    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18\": container with ID starting with 6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18 not found: ID does not exist" containerID="6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18"
Sep 16 12:41:30.209034 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.208956    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18"} err="failed to get container status \"6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18\": rpc error: code = NotFound desc = could not find container \"6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18\": container with ID starting with 6ce181a8f5e596a4062dfa294e81a24f286824c3e5ed931cde5d864a17104d18 not found: ID does not exist"
Sep 16 12:41:30.211022 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.211002    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:41:30.211391 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.211373    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12ada3f-c5e3-41ed-a936-680ff68e6e13" containerName="registry"
Sep 16 12:41:30.211391 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.211393    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12ada3f-c5e3-41ed-a936-680ff68e6e13" containerName="registry"
Sep 16 12:41:30.211543 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.211455    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12ada3f-c5e3-41ed-a936-680ff68e6e13" containerName="registry"
Sep 16 12:41:30.216468 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.216448    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.219002 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.218981    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 12:41:30.219134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219081    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 12:41:30.219134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219081    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 12:41:30.219292 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219270    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 12:41:30.219564 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219544    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 12:41:30.219661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219607    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 12:41:30.219661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219607    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 12:41:30.219862 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219847    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vmjl6\""
Sep 16 12:41:30.219922 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219879    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 12:41:30.219971 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.219925    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 12:41:30.228118 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.228094    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b844f9c45-d6jnb"]
Sep 16 12:41:30.237523 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.237500    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:41:30.238842 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.238823    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b844f9c45-d6jnb"]
Sep 16 12:41:30.281367 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281304    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281367 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbvrm\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-kube-api-access-wbvrm\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281589 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281459    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281589 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281680 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281680 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281635    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281764 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281764 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281719    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281859 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281809    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-web-config\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281901 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281901 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281884    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281984 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281924    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-out\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.281984 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.281952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.382796 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-web-config\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.382922 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382817    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.382922 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.382922 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-out\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.382922 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.382976    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbvrm\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-kube-api-access-wbvrm\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383003    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383042    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383070    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383463 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383463 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383141    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383562 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.383529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.383663 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:41:30.383645    2559 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 12:41:30.383732 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:41:30.383720    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls podName:7ee2f1ec-8e02-48c5-8c4c-693fab078f3e nodeName:}" failed. No retries permitted until 2026-09-16 12:41:30.883698748 +0000 UTC m=+192.967141746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e") : secret "alertmanager-main-tls" not found
Sep 16 12:41:30.384032 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:41:30.383859    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle podName:7ee2f1ec-8e02-48c5-8c4c-693fab078f3e nodeName:}" failed. No retries permitted until 2026-09-16 12:41:30.883840261 +0000 UTC m=+192.967283253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e") : configmap references non-existent config key: ca-bundle.crt
Sep 16 12:41:30.384294 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.384235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.386757 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.386735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-web-config\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.387538 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.387494    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-out\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.388198 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.388124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.388198 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.388142    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.388480 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.388448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.388578 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.388472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-volume\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.389039 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.388980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.390149 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.390129    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.398457 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.398433    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbvrm\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-kube-api-access-wbvrm\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.415089 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.415055    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12ada3f-c5e3-41ed-a936-680ff68e6e13" path="/var/lib/kubelet/pods/d12ada3f-c5e3-41ed-a936-680ff68e6e13/volumes"
Sep 16 12:41:30.887409 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.887373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.887588 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.887418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.888130 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.888111    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:30.889845 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:30.889827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:31.125873 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:31.125842    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:41:31.205458 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:31.205431    2559 generic.go:356] "Generic (PLEG): container finished" podID="664066a5-a1d8-4be9-ad02-ab9a485e70b0" containerID="5f880f79f642dd421d6735d249bc6d21d6ce51c2e2835ea7c04b9c0fac509a20" exitCode=0
Sep 16 12:41:31.205766 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:31.205474    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9dw2" event={"ID":"664066a5-a1d8-4be9-ad02-ab9a485e70b0","Type":"ContainerDied","Data":"5f880f79f642dd421d6735d249bc6d21d6ce51c2e2835ea7c04b9c0fac509a20"}
Sep 16 12:41:31.406850 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:31.406820    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:41:31.410280 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:31.410252    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee2f1ec_8e02_48c5_8c4c_693fab078f3e.slice/crio-c1dd8aa85809ebd07c6ffede6ecbc30c80a79346c15da4f3a2ff6c2bbbfcee23 WatchSource:0}: Error finding container c1dd8aa85809ebd07c6ffede6ecbc30c80a79346c15da4f3a2ff6c2bbbfcee23: Status 404 returned error can't find the container with id c1dd8aa85809ebd07c6ffede6ecbc30c80a79346c15da4f3a2ff6c2bbbfcee23
Sep 16 12:41:32.211340 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:32.211282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9dw2" event={"ID":"664066a5-a1d8-4be9-ad02-ab9a485e70b0","Type":"ContainerStarted","Data":"883de4c048fc5aea4c975062ebf5d78f0e400cf2d3ff4bf88edac8148fd49b49"}
Sep 16 12:41:32.211787 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:32.211371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9dw2" event={"ID":"664066a5-a1d8-4be9-ad02-ab9a485e70b0","Type":"ContainerStarted","Data":"42d918cc66b3794a8c4586b0417c661debd7c49743963112a22aeb6e1520fb02"}
Sep 16 12:41:32.212491 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:32.212469    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"c1dd8aa85809ebd07c6ffede6ecbc30c80a79346c15da4f3a2ff6c2bbbfcee23"}
Sep 16 12:41:32.235057 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:32.235007    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b9dw2" podStartSLOduration=2.17487648 podStartE2EDuration="3.234991311s" totalImagesPullingTime="990.998375ms" totalInitContainerRuntime="69.116456ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:32.234311109 +0000 UTC m=+194.317754143" watchObservedRunningTime="2026-09-16 12:41:32.234991311 +0000 UTC m=+194.318434358"
Sep 16 12:41:33.509182 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.509150    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5dcb4547f6-vdppr"]
Sep 16 12:41:33.512887 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.512867    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.516347 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.516307    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 12:41:33.516431 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.516307    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 12:41:33.517291 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.517187    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s6ljs\""
Sep 16 12:41:33.517291 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.517226    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:41:33.517291 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.517258    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 12:41:33.517291 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.517264    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cpcc1447nrds8\""
Sep 16 12:41:33.524605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.524586    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dcb4547f6-vdppr"]
Sep 16 12:41:33.609466 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609431    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e54a105a-2236-4841-8143-49986fb19dfb-metrics-server-audit-profiles\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.609645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609509    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw8rl\" (UniqueName: \"kubernetes.io/projected/e54a105a-2236-4841-8143-49986fb19dfb-kube-api-access-xw8rl\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.609645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609537    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.609645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609566    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e54a105a-2236-4841-8143-49986fb19dfb-audit-log\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.609645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-secret-metrics-server-tls\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.609645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e54a105a-2236-4841-8143-49986fb19dfb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.609961 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.609675    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-client-ca-bundle\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.710806 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.710776    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-client-ca-bundle\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.710806 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.710812    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e54a105a-2236-4841-8143-49986fb19dfb-metrics-server-audit-profiles\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711019 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.710861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw8rl\" (UniqueName: \"kubernetes.io/projected/e54a105a-2236-4841-8143-49986fb19dfb-kube-api-access-xw8rl\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711060 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.711041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711110 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.711078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e54a105a-2236-4841-8143-49986fb19dfb-audit-log\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711209 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.711186    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-secret-metrics-server-tls\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711286 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.711257    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e54a105a-2236-4841-8143-49986fb19dfb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711437 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.711410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e54a105a-2236-4841-8143-49986fb19dfb-audit-log\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.711846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.711828    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e54a105a-2236-4841-8143-49986fb19dfb-metrics-server-audit-profiles\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.712060 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.712040    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e54a105a-2236-4841-8143-49986fb19dfb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.713430 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.713410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-client-ca-bundle\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.713598 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.713582    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-secret-metrics-server-tls\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.713688 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.713670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e54a105a-2236-4841-8143-49986fb19dfb-secret-metrics-server-client-certs\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.720071 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.720049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw8rl\" (UniqueName: \"kubernetes.io/projected/e54a105a-2236-4841-8143-49986fb19dfb-kube-api-access-xw8rl\") pod \"metrics-server-5dcb4547f6-vdppr\" (UID: \"e54a105a-2236-4841-8143-49986fb19dfb\") " pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.823798 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.823772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:33.903613 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.903581    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-549b995bd5-t4z24"]
Sep 16 12:41:33.906821 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.906803    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:33.909695 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.909672    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:41:33.909695 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.909680    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:41:33.910181 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.910169    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:41:33.910392 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.910377    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:41:33.910709 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.910668    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:41:33.910709 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.910684    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjnxd\""
Sep 16 12:41:33.911227 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.911207    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:41:33.911361 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.911345    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:41:33.915977 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.915959    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:41:33.927657 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:33.927636    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b995bd5-t4z24"]
Sep 16 12:41:34.013861 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.013782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-serving-cert\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.013861 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.013820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-oauth-serving-cert\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.013861 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.013860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq68s\" (UniqueName: \"kubernetes.io/projected/187ecd0c-7dab-4a03-945a-71566028553b-kube-api-access-cq68s\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.014065 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.013886    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-oauth-config\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.014065 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.013980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-service-ca\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.014065 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.014008    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-console-config\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.014165 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.014075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-trusted-ca-bundle\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.114759 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114724    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq68s\" (UniqueName: \"kubernetes.io/projected/187ecd0c-7dab-4a03-945a-71566028553b-kube-api-access-cq68s\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.114759 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114767    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-oauth-config\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115014 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114823    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-service-ca\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115014 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-console-config\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115014 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114905    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-trusted-ca-bundle\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115014 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-serving-cert\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115014 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.114973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-oauth-serving-cert\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115763 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.115735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-oauth-serving-cert\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115859 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.115748    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-console-config\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115937 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.115923    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-trusted-ca-bundle\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.115976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.115954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-service-ca\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.117622 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.117602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-oauth-config\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.117841 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.117823    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-serving-cert\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.118727 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.118709    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5dcb4547f6-vdppr"]
Sep 16 12:41:34.121854 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:34.121831    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54a105a_2236_4841_8143_49986fb19dfb.slice/crio-efe239c88ed9b0786afbf8779f497a31e1152f22cd28d1ee176ae219bba01ce8 WatchSource:0}: Error finding container efe239c88ed9b0786afbf8779f497a31e1152f22cd28d1ee176ae219bba01ce8: Status 404 returned error can't find the container with id efe239c88ed9b0786afbf8779f497a31e1152f22cd28d1ee176ae219bba01ce8
Sep 16 12:41:34.127670 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.127650    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq68s\" (UniqueName: \"kubernetes.io/projected/187ecd0c-7dab-4a03-945a-71566028553b-kube-api-access-cq68s\") pod \"console-549b995bd5-t4z24\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") " pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.215879 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.215848    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:34.221469 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.221446    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c" exitCode=0
Sep 16 12:41:34.221586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.221514    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"}
Sep 16 12:41:34.222730 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.222709    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr" event={"ID":"e54a105a-2236-4841-8143-49986fb19dfb","Type":"ContainerStarted","Data":"efe239c88ed9b0786afbf8779f497a31e1152f22cd28d1ee176ae219bba01ce8"}
Sep 16 12:41:34.494373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:34.494344    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-549b995bd5-t4z24"]
Sep 16 12:41:34.497903 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:34.497875    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187ecd0c_7dab_4a03_945a_71566028553b.slice/crio-b84eac80444d4f19dff6068aae072f578cf82bf0aba7c6f91e73f8375fdfa126 WatchSource:0}: Error finding container b84eac80444d4f19dff6068aae072f578cf82bf0aba7c6f91e73f8375fdfa126: Status 404 returned error can't find the container with id b84eac80444d4f19dff6068aae072f578cf82bf0aba7c6f91e73f8375fdfa126
Sep 16 12:41:35.231474 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.231438    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b995bd5-t4z24" event={"ID":"187ecd0c-7dab-4a03-945a-71566028553b","Type":"ContainerStarted","Data":"b84eac80444d4f19dff6068aae072f578cf82bf0aba7c6f91e73f8375fdfa126"}
Sep 16 12:41:35.465597 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.465558    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:41:35.469956 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.469926    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.474587 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.474427    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 12:41:35.474587 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.474498    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 12:41:35.475025 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475000    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4c0lpblto6hv5\""
Sep 16 12:41:35.475025 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475021    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 12:41:35.475233 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475216    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x5wdw\""
Sep 16 12:41:35.475291 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 12:41:35.475517 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475395    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 12:41:35.475733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475709    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 12:41:35.475830 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475801    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 12:41:35.476046 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475912    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 12:41:35.476046 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.475921    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 12:41:35.476046 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.476029    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 12:41:35.476775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.476632    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 12:41:35.476775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.476651    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 12:41:35.477223 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.477186    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 12:41:35.494595 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.494475    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 12:41:35.510480 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.510453    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.631818    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.631878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.631909    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.631939    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-config-out\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.631963    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.631988    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632015    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-config\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632048    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r782s\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-kube-api-access-r782s\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632131    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-web-config\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632358    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632444    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632493    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632524    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632548    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.632751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.632589    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733493    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733523    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-config-out\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733624    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-config\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733657    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r782s\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-kube-api-access-r782s\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733683    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.733708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-web-config\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733784    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733916    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.733971    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.734007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.734111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.734062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.735590 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.735379    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.735590 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.735434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.735590 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.735437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.735792 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.735714    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.736224 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.736202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.737824 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.737500    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.737930 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.737846    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.738384 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.738358    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.740347 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.740301    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.740526 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.740502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.740645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.740514    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.740733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.740693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-web-config\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.740930 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.740885    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.741104 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.741053    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.741249 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.741225    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-config-out\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.741356 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.741290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-config\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.742608 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.742550    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.743269 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.743224    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.743654 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.743609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.744480 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.744460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r782s\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-kube-api-access-r782s\") pod \"prometheus-k8s-0\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:35.781954 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:35.781879    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:36.238663 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:36.238629    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"}
Sep 16 12:41:36.238988 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:36.238715    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"}
Sep 16 12:41:36.240582 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:36.240554    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr" event={"ID":"e54a105a-2236-4841-8143-49986fb19dfb","Type":"ContainerStarted","Data":"42d8165706ad187e574b30c3181796b0bb9c579d8d8040c8cdd1c6f97c4ed4fd"}
Sep 16 12:41:36.262023 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:36.261698    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr" podStartSLOduration=1.377832565 podStartE2EDuration="3.261674671s" totalImagesPullingTime="1.883842106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:36.260190547 +0000 UTC m=+198.343633580" watchObservedRunningTime="2026-09-16 12:41:36.261674671 +0000 UTC m=+198.345117687"
Sep 16 12:41:36.297627 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:36.297461    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:41:37.248106 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:37.248061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"}
Sep 16 12:41:37.309078 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:41:37.309042    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76790bcb_9ada_4d42_8622_2701acb5e83e.slice/crio-1be06da3a245e614c8a887faef0ba0dc56ed1caa223819c32f55927571e90ebb WatchSource:0}: Error finding container 1be06da3a245e614c8a887faef0ba0dc56ed1caa223819c32f55927571e90ebb: Status 404 returned error can't find the container with id 1be06da3a245e614c8a887faef0ba0dc56ed1caa223819c32f55927571e90ebb
Sep 16 12:41:38.253365 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.253330    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b995bd5-t4z24" event={"ID":"187ecd0c-7dab-4a03-945a-71566028553b","Type":"ContainerStarted","Data":"8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd"}
Sep 16 12:41:38.256881 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.256855    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"}
Sep 16 12:41:38.256993 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.256928    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"}
Sep 16 12:41:38.256993 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.256965    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerStarted","Data":"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"}
Sep 16 12:41:38.258579 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.258557    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05" exitCode=0
Sep 16 12:41:38.258660 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.258589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"}
Sep 16 12:41:38.258660 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.258628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"1be06da3a245e614c8a887faef0ba0dc56ed1caa223819c32f55927571e90ebb"}
Sep 16 12:41:38.274022 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.273973    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-549b995bd5-t4z24" podStartSLOduration=2.411750816 podStartE2EDuration="5.273955326s" totalImagesPullingTime="2.86220451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:38.273709019 +0000 UTC m=+200.357152058" watchObservedRunningTime="2026-09-16 12:41:38.273955326 +0000 UTC m=+200.357398342"
Sep 16 12:41:38.334504 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:38.334456    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.504710539 podStartE2EDuration="8.334437798s" totalImagesPullingTime="4.806836839s" totalInitContainerRuntime="22.89042ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:38.332473595 +0000 UTC m=+200.415916640" watchObservedRunningTime="2026-09-16 12:41:38.334437798 +0000 UTC m=+200.417880847"
Sep 16 12:41:41.276711 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:41.276672    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"}
Sep 16 12:41:41.277071 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:41.276739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"}
Sep 16 12:41:43.287345 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:43.287297    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"}
Sep 16 12:41:43.287782 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:43.287393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"}
Sep 16 12:41:43.287782 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:43.287436    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"}
Sep 16 12:41:43.287782 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:43.287484    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerStarted","Data":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"}
Sep 16 12:41:43.321581 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:43.321485    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.898906254 podStartE2EDuration="8.321468867s" totalImagesPullingTime="4.404290256s" totalInitContainerRuntime="18.272357ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:41:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:41:43.31960899 +0000 UTC m=+205.403052013" watchObservedRunningTime="2026-09-16 12:41:43.321468867 +0000 UTC m=+205.404911880"
Sep 16 12:41:44.216751 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:44.216708    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:44.216930 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:44.216769    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:44.221625 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:44.221604    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:44.295388 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:44.295360    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:41:45.297172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:45.297134    2559 generic.go:356] "Generic (PLEG): container finished" podID="f648151d-c582-4e77-95ee-77fef2cf536b" containerID="36fe766eebe52b343ab25eea0412ff58da1418514ff671909620ec5d45f25ef5" exitCode=0
Sep 16 12:41:45.297547 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:45.297205    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq" event={"ID":"f648151d-c582-4e77-95ee-77fef2cf536b","Type":"ContainerDied","Data":"36fe766eebe52b343ab25eea0412ff58da1418514ff671909620ec5d45f25ef5"}
Sep 16 12:41:45.297690 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:45.297674    2559 scope.go:117] "RemoveContainer" containerID="36fe766eebe52b343ab25eea0412ff58da1418514ff671909620ec5d45f25ef5"
Sep 16 12:41:45.782516 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:45.782476    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:41:46.302768 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:46.302736    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-dhktq" event={"ID":"f648151d-c582-4e77-95ee-77fef2cf536b","Type":"ContainerStarted","Data":"0bfea5213cb63aeecd238ae77fb17ef065edefbb531799aa5d6460daba24d0b0"}
Sep 16 12:41:53.824170 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:53.824144    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:41:53.824604 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:41:53.824193    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:42:05.379049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:05.378956    2559 generic.go:356] "Generic (PLEG): container finished" podID="0cf89e64-8aff-4bdd-a473-0e42517666ed" containerID="8a18565645b5ba13198f7f3e5858c5806bab042ffbcfd673abd04ac5ccf871f5" exitCode=0
Sep 16 12:42:05.379049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:05.378999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jtt5w" event={"ID":"0cf89e64-8aff-4bdd-a473-0e42517666ed","Type":"ContainerDied","Data":"8a18565645b5ba13198f7f3e5858c5806bab042ffbcfd673abd04ac5ccf871f5"}
Sep 16 12:42:05.379619 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:05.379363    2559 scope.go:117] "RemoveContainer" containerID="8a18565645b5ba13198f7f3e5858c5806bab042ffbcfd673abd04ac5ccf871f5"
Sep 16 12:42:06.384609 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:06.384571    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-jtt5w" event={"ID":"0cf89e64-8aff-4bdd-a473-0e42517666ed","Type":"ContainerStarted","Data":"bb74c03f313a589ef2b3b6974adafdf370aee70c755070cf8881af96060ace26"}
Sep 16 12:42:13.829343 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:13.829292    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:42:13.833293 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:13.833245    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5dcb4547f6-vdppr"
Sep 16 12:42:30.219957 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:30.219912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:42:30.222356 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:30.222310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9eca049f-d745-4785-b4a2-a85f698c8b3c-metrics-certs\") pod \"network-metrics-daemon-nc67r\" (UID: \"9eca049f-d745-4785-b4a2-a85f698c8b3c\") " pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:42:30.308731 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:30.308702    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b66st\""
Sep 16 12:42:30.316620 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:30.316600    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nc67r"
Sep 16 12:42:30.608889 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:30.608787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nc67r"]
Sep 16 12:42:30.612692 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:42:30.612656    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eca049f_d745_4785_b4a2_a85f698c8b3c.slice/crio-2cdad1af0fb2f85443d8ee754eb2306aca300a01acf0874c6a57321c4284309b WatchSource:0}: Error finding container 2cdad1af0fb2f85443d8ee754eb2306aca300a01acf0874c6a57321c4284309b: Status 404 returned error can't find the container with id 2cdad1af0fb2f85443d8ee754eb2306aca300a01acf0874c6a57321c4284309b
Sep 16 12:42:31.484422 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:31.484381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nc67r" event={"ID":"9eca049f-d745-4785-b4a2-a85f698c8b3c","Type":"ContainerStarted","Data":"2cdad1af0fb2f85443d8ee754eb2306aca300a01acf0874c6a57321c4284309b"}
Sep 16 12:42:32.490042 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:32.490002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nc67r" event={"ID":"9eca049f-d745-4785-b4a2-a85f698c8b3c","Type":"ContainerStarted","Data":"ebed61f85e9fe6461aa03c9a2eff6735ab3bece6ae8b8d806c773cca1e832c2d"}
Sep 16 12:42:32.490473 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:32.490064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nc67r" event={"ID":"9eca049f-d745-4785-b4a2-a85f698c8b3c","Type":"ContainerStarted","Data":"b264337f4c5eb7140e03c70444fb1c7206d3a3947b6876df979ac4a487d88405"}
Sep 16 12:42:32.509485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:32.509431    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nc67r" podStartSLOduration=252.33552032 podStartE2EDuration="4m13.509415996s" totalImagesPullingTime="1.173895676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:32.508173553 +0000 UTC m=+254.591616578" watchObservedRunningTime="2026-09-16 12:42:32.509415996 +0000 UTC m=+254.592859009"
Sep 16 12:42:35.782459 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:35.782422    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:35.801095 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:35.801073    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:36.523522 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:36.523497    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:49.503109 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.503015    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:42:49.503689 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.503415    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="alertmanager" containerID="cri-o://51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e" gracePeriod=120
Sep 16 12:42:49.503689 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.503464    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-metric" containerID="cri-o://c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd" gracePeriod=120
Sep 16 12:42:49.503689 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.503481    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-web" containerID="cri-o://556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7" gracePeriod=120
Sep 16 12:42:49.504070 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.503497    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy" containerID="cri-o://d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947" gracePeriod=120
Sep 16 12:42:49.504586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.504177    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="config-reloader" containerID="cri-o://b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2" gracePeriod=120
Sep 16 12:42:49.504586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:49.504214    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="prom-label-proxy" containerID="cri-o://6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1" gracePeriod=120
Sep 16 12:42:50.560339 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560288    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1" exitCode=0
Sep 16 12:42:50.560339 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560331    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947" exitCode=0
Sep 16 12:42:50.560339 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560344    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2" exitCode=0
Sep 16 12:42:50.560775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560352    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e" exitCode=0
Sep 16 12:42:50.560775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"}
Sep 16 12:42:50.560775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"}
Sep 16 12:42:50.560775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560442    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"}
Sep 16 12:42:50.560775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.560467    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"}
Sep 16 12:42:50.762421 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.762400    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:50.889833 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889736    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-tls-assets\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.889833 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889784    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-out\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.889833 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889816    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-volume\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889840    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-main-db\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889866    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889930    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889954    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-web-config\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.889988    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-cluster-tls-config\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890074    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890100    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-metrics-client-ca\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890125    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbvrm\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-kube-api-access-wbvrm\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890149    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-web\") pod \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\" (UID: \"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e\") "
Sep 16 12:42:50.890723 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890262    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:50.890723 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890454    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-main-db\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.890825 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.890806    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:50.891177 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.891149    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:50.893777 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.893750    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.894648 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.894492    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:50.894648 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.894503    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-volume" (OuterVolumeSpecName: "config-volume") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.894648 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.894546    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-out" (OuterVolumeSpecName: "config-out") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:50.894648 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.894560    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.894648 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.894618    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.895357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.895338    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.896203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.896180    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-kube-api-access-wbvrm" (OuterVolumeSpecName: "kube-api-access-wbvrm") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "kube-api-access-wbvrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:50.898350 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.898307    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.904589 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.904569    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-web-config" (OuterVolumeSpecName: "web-config") pod "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" (UID: "7ee2f1ec-8e02-48c5-8c4c-693fab078f3e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:50.991696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991656    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-tls-assets\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991689    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-out\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991698    2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991708    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-main-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991718    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991728    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-web-config\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991736    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991747    2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-cluster-tls-config\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991755    2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991763    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-metrics-client-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991771    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbvrm\" (UniqueName: \"kubernetes.io/projected/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-kube-api-access-wbvrm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:50.991998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:50.991778    2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:51.567902 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.567867    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd" exitCode=0
Sep 16 12:42:51.567902 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.567897    2559 generic.go:356] "Generic (PLEG): container finished" podID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerID="556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7" exitCode=0
Sep 16 12:42:51.568310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.567935    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.568310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.567948    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"}
Sep 16 12:42:51.568310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.568022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"}
Sep 16 12:42:51.568310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.568052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7ee2f1ec-8e02-48c5-8c4c-693fab078f3e","Type":"ContainerDied","Data":"c1dd8aa85809ebd07c6ffede6ecbc30c80a79346c15da4f3a2ff6c2bbbfcee23"}
Sep 16 12:42:51.568310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.568090    2559 scope.go:117] "RemoveContainer" containerID="6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"
Sep 16 12:42:51.585467 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.585450    2559 scope.go:117] "RemoveContainer" containerID="c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"
Sep 16 12:42:51.592680 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.592661    2559 scope.go:117] "RemoveContainer" containerID="d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"
Sep 16 12:42:51.599780 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.599760    2559 scope.go:117] "RemoveContainer" containerID="556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"
Sep 16 12:42:51.607554 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.607530    2559 scope.go:117] "RemoveContainer" containerID="b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"
Sep 16 12:42:51.608447 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.608424    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:42:51.615090 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.615053    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:42:51.616183 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.616164    2559 scope.go:117] "RemoveContainer" containerID="51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"
Sep 16 12:42:51.623886 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.623743    2559 scope.go:117] "RemoveContainer" containerID="9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"
Sep 16 12:42:51.630953 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.630928    2559 scope.go:117] "RemoveContainer" containerID="6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"
Sep 16 12:42:51.631242 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.631222    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1\": container with ID starting with 6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1 not found: ID does not exist" containerID="6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"
Sep 16 12:42:51.631339 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.631255    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"} err="failed to get container status \"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1\": rpc error: code = NotFound desc = could not find container \"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1\": container with ID starting with 6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1 not found: ID does not exist"
Sep 16 12:42:51.631339 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.631275    2559 scope.go:117] "RemoveContainer" containerID="c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"
Sep 16 12:42:51.631561 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.631545    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd\": container with ID starting with c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd not found: ID does not exist" containerID="c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"
Sep 16 12:42:51.631613 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.631565    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"} err="failed to get container status \"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd\": rpc error: code = NotFound desc = could not find container \"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd\": container with ID starting with c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd not found: ID does not exist"
Sep 16 12:42:51.631613 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.631583    2559 scope.go:117] "RemoveContainer" containerID="d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"
Sep 16 12:42:51.631856 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.631838    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947\": container with ID starting with d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947 not found: ID does not exist" containerID="d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"
Sep 16 12:42:51.631904 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.631862    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"} err="failed to get container status \"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947\": rpc error: code = NotFound desc = could not find container \"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947\": container with ID starting with d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947 not found: ID does not exist"
Sep 16 12:42:51.631904 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.631877    2559 scope.go:117] "RemoveContainer" containerID="556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"
Sep 16 12:42:51.632121 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.632105    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7\": container with ID starting with 556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7 not found: ID does not exist" containerID="556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"
Sep 16 12:42:51.632167 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632125    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"} err="failed to get container status \"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7\": rpc error: code = NotFound desc = could not find container \"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7\": container with ID starting with 556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7 not found: ID does not exist"
Sep 16 12:42:51.632167 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632137    2559 scope.go:117] "RemoveContainer" containerID="b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"
Sep 16 12:42:51.632377 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.632353    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2\": container with ID starting with b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2 not found: ID does not exist" containerID="b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"
Sep 16 12:42:51.632421 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632380    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"} err="failed to get container status \"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2\": rpc error: code = NotFound desc = could not find container \"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2\": container with ID starting with b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2 not found: ID does not exist"
Sep 16 12:42:51.632421 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632394    2559 scope.go:117] "RemoveContainer" containerID="51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"
Sep 16 12:42:51.632626 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.632608    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e\": container with ID starting with 51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e not found: ID does not exist" containerID="51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"
Sep 16 12:42:51.632666 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632631    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"} err="failed to get container status \"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e\": rpc error: code = NotFound desc = could not find container \"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e\": container with ID starting with 51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e not found: ID does not exist"
Sep 16 12:42:51.632666 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632642    2559 scope.go:117] "RemoveContainer" containerID="9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"
Sep 16 12:42:51.632848 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:51.632834    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c\": container with ID starting with 9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c not found: ID does not exist" containerID="9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"
Sep 16 12:42:51.632884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632850    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"} err="failed to get container status \"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c\": rpc error: code = NotFound desc = could not find container \"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c\": container with ID starting with 9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c not found: ID does not exist"
Sep 16 12:42:51.632884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.632861    2559 scope.go:117] "RemoveContainer" containerID="6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"
Sep 16 12:42:51.633101 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633083    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1"} err="failed to get container status \"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1\": rpc error: code = NotFound desc = could not find container \"6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1\": container with ID starting with 6653b38f98f72f2409738c359e7013854363613fbe495a26d657c0db8e0b39e1 not found: ID does not exist"
Sep 16 12:42:51.633151 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633102    2559 scope.go:117] "RemoveContainer" containerID="c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"
Sep 16 12:42:51.633428 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633410    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd"} err="failed to get container status \"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd\": rpc error: code = NotFound desc = could not find container \"c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd\": container with ID starting with c61fd45b03554a6fb0732203b097536fabbf9e43ace91c3058e206115722b3dd not found: ID does not exist"
Sep 16 12:42:51.633477 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633429    2559 scope.go:117] "RemoveContainer" containerID="d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"
Sep 16 12:42:51.633656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633640    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947"} err="failed to get container status \"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947\": rpc error: code = NotFound desc = could not find container \"d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947\": container with ID starting with d95a3f839c17dd62942a21ec663e68e4d938432a3db1727badae56c9074fe947 not found: ID does not exist"
Sep 16 12:42:51.633695 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633657    2559 scope.go:117] "RemoveContainer" containerID="556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"
Sep 16 12:42:51.633879 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633861    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7"} err="failed to get container status \"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7\": rpc error: code = NotFound desc = could not find container \"556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7\": container with ID starting with 556bb60edd65c19e8d0867bc4ea3ad66dc5738d5bb2f85cca8129970437e55c7 not found: ID does not exist"
Sep 16 12:42:51.633928 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.633880    2559 scope.go:117] "RemoveContainer" containerID="b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"
Sep 16 12:42:51.634115 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.634100    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2"} err="failed to get container status \"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2\": rpc error: code = NotFound desc = could not find container \"b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2\": container with ID starting with b30e25977491c6b9daa4c78ccbc95eeae89e752792f031235a67d952ef0992a2 not found: ID does not exist"
Sep 16 12:42:51.634151 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.634117    2559 scope.go:117] "RemoveContainer" containerID="51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"
Sep 16 12:42:51.634352 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.634314    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e"} err="failed to get container status \"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e\": rpc error: code = NotFound desc = could not find container \"51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e\": container with ID starting with 51bbcd67f48466c684c1c60eb5eb585353e5a81c2be0e7ca2971f4633bc3231e not found: ID does not exist"
Sep 16 12:42:51.634401 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.634352    2559 scope.go:117] "RemoveContainer" containerID="9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"
Sep 16 12:42:51.634570 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.634549    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c"} err="failed to get container status \"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c\": rpc error: code = NotFound desc = could not find container \"9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c\": container with ID starting with 9d5bd82045c426ce1bd5731e64db2bbbfa3a40a926c1b9b08c3b74a0acb1e20c not found: ID does not exist"
Sep 16 12:42:51.647894 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.647865    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:42:51.648184 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648173    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="init-config-reloader"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648186    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="init-config-reloader"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648194    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648200    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648207    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="prom-label-proxy"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648212    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="prom-label-proxy"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648220    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-metric"
Sep 16 12:42:51.648226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648225    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-metric"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648231    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="alertmanager"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648235    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="alertmanager"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648241    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="config-reloader"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648246    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="config-reloader"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648254    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-web"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648258    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-web"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648305    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648313    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="alertmanager"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648338    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-web"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648345    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="prom-label-proxy"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648351    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="config-reloader"
Sep 16 12:42:51.648442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.648357    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" containerName="kube-rbac-proxy-metric"
Sep 16 12:42:51.653233 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.653216    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.655777 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.655758    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 12:42:51.655886 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.655757    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 12:42:51.655886 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.655854    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 12:42:51.656057 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.656040    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vmjl6\""
Sep 16 12:42:51.656127 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.656110    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 12:42:51.656185 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.656127    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 12:42:51.656236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.656213    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 12:42:51.656293 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.656237    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 12:42:51.656363 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.656341    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 12:42:51.661761 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.661737    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 12:42:51.668978 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.668955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:42:51.699114 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699075    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699114 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699153    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-config-volume\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699168    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-config-out\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699206    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699220    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699295    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699339    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699355    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzx8\" (UniqueName: \"kubernetes.io/projected/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-kube-api-access-cfzx8\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699677 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-web-config\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.699677 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.699392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800592 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800597    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800614    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzx8\" (UniqueName: \"kubernetes.io/projected/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-kube-api-access-cfzx8\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800634    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-web-config\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800655    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800710    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.800753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800755    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801050 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-config-volume\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801050 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.800955    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-config-out\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801050 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.801001    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801050 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.801034    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.801056    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801240 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.801090    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801621 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.801528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.801621 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.801548    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.803872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.803897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-config-volume\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804031    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804105    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804155    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804501 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804274    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-config-out\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804620 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-web-config\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804629    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.804795 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.804776    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.805889 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.805872    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.810113 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.810093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzx8\" (UniqueName: \"kubernetes.io/projected/92c5fe5b-b7ad-4039-8668-e74bf06d7d73-kube-api-access-cfzx8\") pod \"alertmanager-main-0\" (UID: \"92c5fe5b-b7ad-4039-8668-e74bf06d7d73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:51.962863 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:51.962830    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:42:52.246266 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:52.246171    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:42:52.249297 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:42:52.249267    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c5fe5b_b7ad_4039_8668_e74bf06d7d73.slice/crio-68b6dc3288012d1855032b884d5081a51e9517cfc3574699e88ae6777323d6ea WatchSource:0}: Error finding container 68b6dc3288012d1855032b884d5081a51e9517cfc3574699e88ae6777323d6ea: Status 404 returned error can't find the container with id 68b6dc3288012d1855032b884d5081a51e9517cfc3574699e88ae6777323d6ea
Sep 16 12:42:52.410679 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:52.410644    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee2f1ec-8e02-48c5-8c4c-693fab078f3e" path="/var/lib/kubelet/pods/7ee2f1ec-8e02-48c5-8c4c-693fab078f3e/volumes"
Sep 16 12:42:52.580288 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:52.580186    2559 generic.go:356] "Generic (PLEG): container finished" podID="92c5fe5b-b7ad-4039-8668-e74bf06d7d73" containerID="be6d99fc6ff0767bd6a66c2036af3841e523149a557a0bbb1cebb28cf6e3263b" exitCode=0
Sep 16 12:42:52.580288 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:52.580255    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerDied","Data":"be6d99fc6ff0767bd6a66c2036af3841e523149a557a0bbb1cebb28cf6e3263b"}
Sep 16 12:42:52.580699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:52.580371    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"68b6dc3288012d1855032b884d5081a51e9517cfc3574699e88ae6777323d6ea"}
Sep 16 12:42:53.588558 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.588525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"d4eed13a0a2b76068c48ef1c7c083fef563181e96d779fd38172f35a5fd98815"}
Sep 16 12:42:53.588923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.588587    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"b0b93e389209e62a8ffffd3df4ed128917370de00304b6021c6f41de98244af6"}
Sep 16 12:42:53.588923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.588618    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"697bab0c07d44c3c2fde2b49133e26bdff6204c558980af3f764018399f0c412"}
Sep 16 12:42:53.588923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.588643    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"065ad01cfee4ac552bf098580628976aa4130925c9d1640bcd47aefb3811c9c9"}
Sep 16 12:42:53.588923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.588668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"850f7b12ed42a74074c1dd63f184daa047660e4818d145d37e77bf3c8dcc178e"}
Sep 16 12:42:53.588923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.588702    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92c5fe5b-b7ad-4039-8668-e74bf06d7d73","Type":"ContainerStarted","Data":"1b9ac42fbc5a7539cd4fed47b9e9d0ac56c3b35aec5ec618f339cec4dad93f0b"}
Sep 16 12:42:53.629110 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.629054    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.614430451 podStartE2EDuration="2.629035059s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.604608ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:53.627687432 +0000 UTC m=+275.711130453" watchObservedRunningTime="2026-09-16 12:42:53.629035059 +0000 UTC m=+275.712478116"
Sep 16 12:42:53.893821 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.893742    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:42:53.894054 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.894017    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="prometheus" containerID="cri-o://4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1" gracePeriod=600
Sep 16 12:42:53.894054 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.894050    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249" gracePeriod=600
Sep 16 12:42:53.894276 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.894074    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="thanos-sidecar" containerID="cri-o://9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d" gracePeriod=600
Sep 16 12:42:53.894276 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.894027    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy" containerID="cri-o://71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7" gracePeriod=600
Sep 16 12:42:53.894276 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.894108    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="config-reloader" containerID="cri-o://33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123" gracePeriod=600
Sep 16 12:42:53.894276 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:53.894081    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-web" containerID="cri-o://9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d" gracePeriod=600
Sep 16 12:42:54.137080 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.137054    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.221438 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221406    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-config-out\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221462    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-serving-certs-ca-bundle\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221491    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-config\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-0\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221532    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-1\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221561    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221583    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-grpc-tls\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.221605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.221604    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-thanos-prometheus-http-client-file\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222070 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222020    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.222180 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222139    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-kube-rbac-proxy\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222284    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.222439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-metrics-client-ca\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222426    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-tls-assets\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222591 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222459    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-db\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222591 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222486    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-metrics-client-certs\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222591 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222535    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-2\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222591 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222572    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r782s\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-kube-api-access-r782s\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222598    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-web-config\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222627    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-kubelet-serving-ca-bundle\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222653    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-trusted-ca-bundle\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222693    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.222801 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222719    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-tls\") pod \"76790bcb-9ada-4d42-8622-2701acb5e83e\" (UID: \"76790bcb-9ada-4d42-8622-2701acb5e83e\") "
Sep 16 12:42:54.223032 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.222997    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.223032 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.223018    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.223362 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.223334    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.223775 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.223738    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.225130 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225033    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.225130 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225037    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.225130 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225100    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-config-out" (OuterVolumeSpecName: "config-out") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:54.225130 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225112    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.225687 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225658    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.225899 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225871    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-config" (OuterVolumeSpecName: "config") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.225973 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.225914    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:42:54.226475 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.226425    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.226673 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.226647    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.226784 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.226763    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.226902 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.226882    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:42:54.226982 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.226894    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.227651 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.227630    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:54.228272 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.228242    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.228355 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.228290    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-kube-api-access-r782s" (OuterVolumeSpecName: "kube-api-access-r782s") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "kube-api-access-r782s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:42:54.237841 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.237815    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-web-config" (OuterVolumeSpecName: "web-config") pod "76790bcb-9ada-4d42-8622-2701acb5e83e" (UID: "76790bcb-9ada-4d42-8622-2701acb5e83e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:42:54.323762 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323730    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323762 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323758    2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-grpc-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323772    2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323783    2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-kube-rbac-proxy\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323792    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-metrics-client-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323801    2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-tls-assets\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323810    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-db\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323817    2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-metrics-client-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323828    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323836    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r782s\" (UniqueName: \"kubernetes.io/projected/76790bcb-9ada-4d42-8622-2701acb5e83e-kube-api-access-r782s\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323844    2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-web-config\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323852    2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323860    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323868    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323877    2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323884    2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76790bcb-9ada-4d42-8622-2701acb5e83e-config-out\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323891    2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/76790bcb-9ada-4d42-8622-2701acb5e83e-config\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.323966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.323899    2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76790bcb-9ada-4d42-8622-2701acb5e83e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:42:54.596717 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596682    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249" exitCode=0
Sep 16 12:42:54.596717 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596706    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7" exitCode=0
Sep 16 12:42:54.596717 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596714    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d" exitCode=0
Sep 16 12:42:54.596717 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596719    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d" exitCode=0
Sep 16 12:42:54.596717 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596726    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123" exitCode=0
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596734    2559 generic.go:356] "Generic (PLEG): container finished" podID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1" exitCode=0
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596763    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"}
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596831    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"}
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596771    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596869    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.596859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"}
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.597057    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"}
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.597104    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"}
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.597147    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"}
Sep 16 12:42:54.597368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.597187    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76790bcb-9ada-4d42-8622-2701acb5e83e","Type":"ContainerDied","Data":"1be06da3a245e614c8a887faef0ba0dc56ed1caa223819c32f55927571e90ebb"}
Sep 16 12:42:54.605811 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.605792    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.613140 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.613120    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.620649 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.620625    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.621498 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.621469    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:42:54.628953 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.628923    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:42:54.629368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.629347    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.637134 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.637115    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.645076 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.645057    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.652490 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.652467    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.652801 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.652780    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.652905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.652811    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"} err="failed to get container status \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist"
Sep 16 12:42:54.652905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.652831    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.653098 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.653081    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.653137 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653104    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"} err="failed to get container status \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist"
Sep 16 12:42:54.653137 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653119    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.653408 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.653390    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.653477 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653414    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"} err="failed to get container status \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist"
Sep 16 12:42:54.653477 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653428    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.653704 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.653686    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.653741 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653708    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"} err="failed to get container status \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist"
Sep 16 12:42:54.653741 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653721    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.653966 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.653951    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.654001 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653969    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"} err="failed to get container status \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist"
Sep 16 12:42:54.654001 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.653981    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.654225 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.654210    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.654270 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654251    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"} err="failed to get container status \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist"
Sep 16 12:42:54.654270 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654264    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.654521 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:42:54.654505    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.654560 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654526    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"} err="failed to get container status \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist"
Sep 16 12:42:54.654560 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654538    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.654757 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654740    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"} err="failed to get container status \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist"
Sep 16 12:42:54.654797 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654759    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.655006 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.654989    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"} err="failed to get container status \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist"
Sep 16 12:42:54.655044 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655008    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.655236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655215    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"} err="failed to get container status \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist"
Sep 16 12:42:54.655311 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655238    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.655483 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655467    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"} err="failed to get container status \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist"
Sep 16 12:42:54.655532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655484    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.655733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655716    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"} err="failed to get container status \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist"
Sep 16 12:42:54.655778 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655734    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.655983 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655965    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"} err="failed to get container status \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist"
Sep 16 12:42:54.655983 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.655982    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.656204 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.656186    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"} err="failed to get container status \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist"
Sep 16 12:42:54.656246 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.656206    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.656451 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.656433    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"} err="failed to get container status \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist"
Sep 16 12:42:54.656497 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.656452    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657014    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"} err="failed to get container status \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657064    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657374    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"} err="failed to get container status \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657401    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657663    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"} err="failed to get container status \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657691    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657937    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"} err="failed to get container status \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.657965    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.658297    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"} err="failed to get container status \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.658345    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.658615    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"} err="failed to get container status \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.658636    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.658864    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"} err="failed to get container status \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.658886    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659131    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"} err="failed to get container status \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659152    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.662082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659410    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"} err="failed to get container status \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659431    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659690    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"} err="failed to get container status \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659717    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659967    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"} err="failed to get container status \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.659988    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.660257    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"} err="failed to get container status \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.660281    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.660577    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"} err="failed to get container status \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.660603    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.660857    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"} err="failed to get container status \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.660876    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661122    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"} err="failed to get container status \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661144    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661404    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"} err="failed to get container status \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661431    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661671    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"} err="failed to get container status \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist"
Sep 16 12:42:54.662976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661692    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.663618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.661945    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"} err="failed to get container status \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist"
Sep 16 12:42:54.663618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.662030    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.663618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.663512    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"} err="failed to get container status \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist"
Sep 16 12:42:54.663618 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.663530    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.663815 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.663792    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"} err="failed to get container status \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist"
Sep 16 12:42:54.663858 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.663818    2559 scope.go:117] "RemoveContainer" containerID="4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"
Sep 16 12:42:54.664124 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664102    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249"} err="failed to get container status \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": rpc error: code = NotFound desc = could not find container \"4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249\": container with ID starting with 4348b13c2be1cb347aed3ba918ae34b1545f419d4d064fbe1260c923101f3249 not found: ID does not exist"
Sep 16 12:42:54.664192 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664126    2559 scope.go:117] "RemoveContainer" containerID="71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"
Sep 16 12:42:54.664428 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664406    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7"} err="failed to get container status \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": rpc error: code = NotFound desc = could not find container \"71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7\": container with ID starting with 71d77a9aa80deef99cbc8e23f3f96c6ecb1cf2aab241bfd87900c43a2e640fd7 not found: ID does not exist"
Sep 16 12:42:54.664485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664429    2559 scope.go:117] "RemoveContainer" containerID="9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"
Sep 16 12:42:54.664707 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664684    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d"} err="failed to get container status \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": rpc error: code = NotFound desc = could not find container \"9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d\": container with ID starting with 9f8671e99d7a4cf142a335f8f13a2caf62c90aa8dd449472a76b9b918cc3f45d not found: ID does not exist"
Sep 16 12:42:54.664771 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664709    2559 scope.go:117] "RemoveContainer" containerID="9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"
Sep 16 12:42:54.664954 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664933    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d"} err="failed to get container status \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": rpc error: code = NotFound desc = could not find container \"9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d\": container with ID starting with 9de66946328fbc045b2cf70e3ca2c9264c203299eb50f56cafdc676c23f65f2d not found: ID does not exist"
Sep 16 12:42:54.664998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664957    2559 scope.go:117] "RemoveContainer" containerID="33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"
Sep 16 12:42:54.664998 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.664973    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:42:54.665263 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665236    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123"} err="failed to get container status \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": rpc error: code = NotFound desc = could not find container \"33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123\": container with ID starting with 33c7c909ede0696a875279f47e408bdaddef6d50eacd1df4260b379aea6d2123 not found: ID does not exist"
Sep 16 12:42:54.665360 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665268    2559 scope.go:117] "RemoveContainer" containerID="4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"
Sep 16 12:42:54.665644 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665617    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1"} err="failed to get container status \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": rpc error: code = NotFound desc = could not find container \"4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1\": container with ID starting with 4a710b215f03c7f4cdc241c933c0effa765a7de47b1f3cdbdc30a69f7b708dc1 not found: ID does not exist"
Sep 16 12:42:54.665710 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665651    2559 scope.go:117] "RemoveContainer" containerID="8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"
Sep 16 12:42:54.665742 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665713    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="thanos-sidecar"
Sep 16 12:42:54.665742 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665731    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="thanos-sidecar"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665742    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665749    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665759    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="init-config-reloader"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665765    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="init-config-reloader"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665793    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-web"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665806    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-web"
Sep 16 12:42:54.665829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665828    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="config-reloader"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665838    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="config-reloader"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665850    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-thanos"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665862    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-thanos"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665899    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="prometheus"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665909    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="prometheus"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665994    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="config-reloader"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.665989    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05"} err="failed to get container status \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": rpc error: code = NotFound desc = could not find container \"8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05\": container with ID starting with 8015d31f4f4cc896d892e1e47c9071acbbaf91a0c879a6a0d36bec144362fb05 not found: ID does not exist"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.666006    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-web"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.666012    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="prometheus"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.666019    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.666024    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="kube-rbac-proxy-thanos"
Sep 16 12:42:54.666052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.666029    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" containerName="thanos-sidecar"
Sep 16 12:42:54.671192 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.671177    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.673862 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.673844    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 12:42:54.674000 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.673984    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 12:42:54.674133 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674118    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x5wdw\""
Sep 16 12:42:54.674231 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674212    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 12:42:54.674284 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674230    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 12:42:54.674284 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674236    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 12:42:54.674454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674367    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 12:42:54.674558 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674543    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 12:42:54.674641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674553    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 12:42:54.674700 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674642    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 12:42:54.674755 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674741    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 12:42:54.674946 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.674932    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 12:42:54.675182 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.675168    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 12:42:54.675279 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.675264    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4c0lpblto6hv5\""
Sep 16 12:42:54.677954 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.677936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 12:42:54.680022 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.680003    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 12:42:54.688548 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.688523    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:42:54.727608 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727570    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727608 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727611    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727630    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d230233-669d-4457-be09-6784a6a5552b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727661    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727755    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d230233-669d-4457-be09-6784a6a5552b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727816    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.727846 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727858    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727874    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxg8f\" (UniqueName: \"kubernetes.io/projected/8d230233-669d-4457-be09-6784a6a5552b-kube-api-access-lxg8f\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727906    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-config\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.727968    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.728139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.728163    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.728188    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728373 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.728361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.728568 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.728396    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830020 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.829935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830020 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.829977    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830020 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830007    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830036    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d230233-669d-4457-be09-6784a6a5552b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830260    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d230233-669d-4457-be09-6784a6a5552b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830369    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830412    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830439    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxg8f\" (UniqueName: \"kubernetes.io/projected/8d230233-669d-4457-be09-6784a6a5552b-kube-api-access-lxg8f\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830465    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-config\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830508    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830545    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830572    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830627    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.830948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.831499 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.831063    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.831682 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.831656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.833529 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.833498    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8d230233-669d-4457-be09-6784a6a5552b-config-out\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.833668 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.833645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.833735 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.833659    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.834081 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834057    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-web-config\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.834174 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834147    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.834174 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.834353 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834309    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.834433 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.834433 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.830466    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.835095 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.835095 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.835095 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.834898    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8d230233-669d-4457-be09-6784a6a5552b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.835300 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.835153    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.836632 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.836612    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-config\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.836733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.836660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8d230233-669d-4457-be09-6784a6a5552b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.836905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.836888    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.836964 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.836894    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8d230233-669d-4457-be09-6784a6a5552b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.844111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.844073    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxg8f\" (UniqueName: \"kubernetes.io/projected/8d230233-669d-4457-be09-6784a6a5552b-kube-api-access-lxg8f\") pod \"prometheus-k8s-0\" (UID: \"8d230233-669d-4457-be09-6784a6a5552b\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:54.981815 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:54.981776    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:42:55.285738 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:55.285704    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 12:42:55.288981 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:42:55.288955    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d230233_669d_4457_be09_6784a6a5552b.slice/crio-2806e19ef255e6085add571c70e6a5d59857f17b12f331d4c7464394df81f5f0 WatchSource:0}: Error finding container 2806e19ef255e6085add571c70e6a5d59857f17b12f331d4c7464394df81f5f0: Status 404 returned error can't find the container with id 2806e19ef255e6085add571c70e6a5d59857f17b12f331d4c7464394df81f5f0
Sep 16 12:42:55.603165 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:55.603075    2559 generic.go:356] "Generic (PLEG): container finished" podID="8d230233-669d-4457-be09-6784a6a5552b" containerID="2c8fa483b5e297a6712ae55ace42bd4f96f5fa1af3873cc8395e145ae54394e7" exitCode=0
Sep 16 12:42:55.603640 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:55.603174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerDied","Data":"2c8fa483b5e297a6712ae55ace42bd4f96f5fa1af3873cc8395e145ae54394e7"}
Sep 16 12:42:55.603640 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:55.603263    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"2806e19ef255e6085add571c70e6a5d59857f17b12f331d4c7464394df81f5f0"}
Sep 16 12:42:56.414614 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.414578    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76790bcb-9ada-4d42-8622-2701acb5e83e" path="/var/lib/kubelet/pods/76790bcb-9ada-4d42-8622-2701acb5e83e/volumes"
Sep 16 12:42:56.611976 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.611942    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"c2f01832d94f22df0b14fe54f6592414d561f3ae893e635d3f1eebfaa077e180"}
Sep 16 12:42:56.612380 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.612003    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"dcec1e4e292dd7ba62f09414eb7e099a305cd7d212bc842567ae39faf13fd3d5"}
Sep 16 12:42:56.612380 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.612029    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"e2c5c69292e3bd9c18a22c2b95d75aa081ba008527c002d5e36b518e7e4498cd"}
Sep 16 12:42:56.612380 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.612051    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"95806766375bae7d8c0f50a6301208883bf32a158acb5df567cb16e603663784"}
Sep 16 12:42:56.612380 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.612075    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"f47b272f15b3e50a3183a7c36c0f5958f02dd2b79e6447245919a2b12c9881f3"}
Sep 16 12:42:56.612380 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.612097    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8d230233-669d-4457-be09-6784a6a5552b","Type":"ContainerStarted","Data":"fe5bc6687fff67e1f6b73a3dd755c174613dd154c09a44e413e36c186642d5f4"}
Sep 16 12:42:56.646333 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:56.646269    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.624652459 podStartE2EDuration="2.646245434s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.592975ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:42:56.644818123 +0000 UTC m=+278.728261140" watchObservedRunningTime="2026-09-16 12:42:56.646245434 +0000 UTC m=+278.729688448"
Sep 16 12:42:59.982660 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:42:59.982620    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:43:09.013728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:09.013697    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b995bd5-t4z24"]
Sep 16 12:43:18.361651 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:18.361607    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:43:34.034021 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.033956    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-549b995bd5-t4z24" podUID="187ecd0c-7dab-4a03-945a-71566028553b" containerName="console" containerID="cri-o://8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd" gracePeriod=15
Sep 16 12:43:34.274548 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.274523    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:43:34.362878 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362791    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-trusted-ca-bundle\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.362878 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362831    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-serving-cert\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.362878 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-oauth-serving-cert\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.363122 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362896    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq68s\" (UniqueName: \"kubernetes.io/projected/187ecd0c-7dab-4a03-945a-71566028553b-kube-api-access-cq68s\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.363122 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-service-ca\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.363122 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362930    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-console-config\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.363122 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.362959    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-oauth-config\") pod \"187ecd0c-7dab-4a03-945a-71566028553b\" (UID: \"187ecd0c-7dab-4a03-945a-71566028553b\") "
Sep 16 12:43:34.363446 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.363418    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:43:34.363530 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.363304    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:43:34.363571 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.363536    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-service-ca" (OuterVolumeSpecName: "service-ca") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:43:34.363571 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.363539    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-console-config" (OuterVolumeSpecName: "console-config") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:43:34.365515 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.365490    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:43:34.365594 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.365532    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/187ecd0c-7dab-4a03-945a-71566028553b-kube-api-access-cq68s" (OuterVolumeSpecName: "kube-api-access-cq68s") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "kube-api-access-cq68s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:43:34.365639 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.365616    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "187ecd0c-7dab-4a03-945a-71566028553b" (UID: "187ecd0c-7dab-4a03-945a-71566028553b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:43:34.464077 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464041    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-oauth-serving-cert\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.464077 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464071    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq68s\" (UniqueName: \"kubernetes.io/projected/187ecd0c-7dab-4a03-945a-71566028553b-kube-api-access-cq68s\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.464077 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464083    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.464307 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464098    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-console-config\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.464307 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464106    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-oauth-config\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.464307 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464115    2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/187ecd0c-7dab-4a03-945a-71566028553b-trusted-ca-bundle\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.464307 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.464123    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/187ecd0c-7dab-4a03-945a-71566028553b-console-serving-cert\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:43:34.767520 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.767476    2559 generic.go:356] "Generic (PLEG): container finished" podID="187ecd0c-7dab-4a03-945a-71566028553b" containerID="8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd" exitCode=0
Sep 16 12:43:34.767520 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.767523    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-549b995bd5-t4z24"
Sep 16 12:43:34.767728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.767557    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b995bd5-t4z24" event={"ID":"187ecd0c-7dab-4a03-945a-71566028553b","Type":"ContainerDied","Data":"8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd"}
Sep 16 12:43:34.767728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.767610    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-549b995bd5-t4z24" event={"ID":"187ecd0c-7dab-4a03-945a-71566028553b","Type":"ContainerDied","Data":"b84eac80444d4f19dff6068aae072f578cf82bf0aba7c6f91e73f8375fdfa126"}
Sep 16 12:43:34.767728 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.767631    2559 scope.go:117] "RemoveContainer" containerID="8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd"
Sep 16 12:43:34.775744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.775723    2559 scope.go:117] "RemoveContainer" containerID="8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd"
Sep 16 12:43:34.776035 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:43:34.776016    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd\": container with ID starting with 8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd not found: ID does not exist" containerID="8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd"
Sep 16 12:43:34.776105 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.776047    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd"} err="failed to get container status \"8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd\": rpc error: code = NotFound desc = could not find container \"8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd\": container with ID starting with 8364add10fcf24ecc01865a02786240046a87ce6d9ea31026697226f1106bdbd not found: ID does not exist"
Sep 16 12:43:34.791827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.791794    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-549b995bd5-t4z24"]
Sep 16 12:43:34.792973 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:34.792953    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-549b995bd5-t4z24"]
Sep 16 12:43:36.410649 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:36.410607    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="187ecd0c-7dab-4a03-945a-71566028553b" path="/var/lib/kubelet/pods/187ecd0c-7dab-4a03-945a-71566028553b/volumes"
Sep 16 12:43:54.982339 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:54.982290    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:43:54.998241 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:54.998202    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:43:55.865103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:43:55.865075    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 12:46:37.585585 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.585553    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"]
Sep 16 12:46:37.586066 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.585916    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="187ecd0c-7dab-4a03-945a-71566028553b" containerName="console"
Sep 16 12:46:37.586066 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.585928    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="187ecd0c-7dab-4a03-945a-71566028553b" containerName="console"
Sep 16 12:46:37.586066 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.585996    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="187ecd0c-7dab-4a03-945a-71566028553b" containerName="console"
Sep 16 12:46:37.588938 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.588923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.591927 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.591901    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:46:37.592049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.591953    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:46:37.592049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.591979    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:46:37.592049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.592005    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:46:37.592221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.592199    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:46:37.592358 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.592267    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xqc5b\""
Sep 16 12:46:37.600424 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.600405    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"]
Sep 16 12:46:37.702358 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.702309    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e00e7b-dab6-4162-ab38-ffd546518889-cert\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.702531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.702423    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e3e00e7b-dab6-4162-ab38-ffd546518889-manager-config\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.702531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.702490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3e00e7b-dab6-4162-ab38-ffd546518889-metrics-cert\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.702531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.702522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmvw\" (UniqueName: \"kubernetes.io/projected/e3e00e7b-dab6-4162-ab38-ffd546518889-kube-api-access-jwmvw\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.803877 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.803841    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e3e00e7b-dab6-4162-ab38-ffd546518889-manager-config\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.803877 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.803881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3e00e7b-dab6-4162-ab38-ffd546518889-metrics-cert\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.804268 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.803901    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmvw\" (UniqueName: \"kubernetes.io/projected/e3e00e7b-dab6-4162-ab38-ffd546518889-kube-api-access-jwmvw\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.804268 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.803952    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e00e7b-dab6-4162-ab38-ffd546518889-cert\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.804514 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.804492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e3e00e7b-dab6-4162-ab38-ffd546518889-manager-config\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.806445 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.806423    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3e00e7b-dab6-4162-ab38-ffd546518889-cert\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.806605 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.806585    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3e00e7b-dab6-4162-ab38-ffd546518889-metrics-cert\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.815874 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.815852    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmvw\" (UniqueName: \"kubernetes.io/projected/e3e00e7b-dab6-4162-ab38-ffd546518889-kube-api-access-jwmvw\") pod \"lws-controller-manager-575fbb86f4-cndrn\" (UID: \"e3e00e7b-dab6-4162-ab38-ffd546518889\") " pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:37.898718 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:37.898652    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:38.175667 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:38.175624    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"]
Sep 16 12:46:38.178713 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:46:38.178687    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e00e7b_dab6_4162_ab38_ffd546518889.slice/crio-2128caba19ad4e453dbaf3d051268d5d5116cf383302f75682e130c74a4394e0 WatchSource:0}: Error finding container 2128caba19ad4e453dbaf3d051268d5d5116cf383302f75682e130c74a4394e0: Status 404 returned error can't find the container with id 2128caba19ad4e453dbaf3d051268d5d5116cf383302f75682e130c74a4394e0
Sep 16 12:46:38.180648 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:38.180632    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:46:38.464089 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:38.464009    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn" event={"ID":"e3e00e7b-dab6-4162-ab38-ffd546518889","Type":"ContainerStarted","Data":"2128caba19ad4e453dbaf3d051268d5d5116cf383302f75682e130c74a4394e0"}
Sep 16 12:46:41.478562 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:41.478529    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn" event={"ID":"e3e00e7b-dab6-4162-ab38-ffd546518889","Type":"ContainerStarted","Data":"c09a038b5c60b7b51cb7ab835be78601108e659cda4cf3d270bfebe3ac2e260c"}
Sep 16 12:46:41.478961 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:41.478658    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:41.499081 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:41.498931    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn" podStartSLOduration=1.5992697919999999 podStartE2EDuration="4.498909545s" totalImagesPullingTime="2.899639753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:41.497913126 +0000 UTC m=+503.581356140" watchObservedRunningTime="2026-09-16 12:46:41.498909545 +0000 UTC m=+503.582352560"
Sep 16 12:46:46.631852 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.631823    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"]
Sep 16 12:46:46.635218 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.635203    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.638726 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:46:46.638654    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret"
Sep 16 12:46:46.638851 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:46:46.638823    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 16 12:46:46.638851 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:46:46.638836    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 16 12:46:46.638960 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:46:46.638872    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-dockercfg-72j5p\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-72j5p\"" type="*v1.Secret"
Sep 16 12:46:46.639108 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:46:46.639084    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="e746e6c0-4127-42e7-b184-d8e90274bf4d" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.639108 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:46:46.639090    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret"
Sep 16 12:46:46.666270 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.666246    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"]
Sep 16 12:46:46.777286 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.777253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e746e6c0-4127-42e7-b184-d8e90274bf4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.777452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.777297    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e746e6c0-4127-42e7-b184-d8e90274bf4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.777452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.777337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e746e6c0-4127-42e7-b184-d8e90274bf4d-manifests\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.777452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.777397    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e746e6c0-4127-42e7-b184-d8e90274bf4d-tmp\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.777452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.777429    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7r8w\" (UniqueName: \"kubernetes.io/projected/e746e6c0-4127-42e7-b184-d8e90274bf4d-kube-api-access-v7r8w\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.878736 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.878706    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e746e6c0-4127-42e7-b184-d8e90274bf4d-tmp\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.878736 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.878740    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7r8w\" (UniqueName: \"kubernetes.io/projected/e746e6c0-4127-42e7-b184-d8e90274bf4d-kube-api-access-v7r8w\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.878941 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.878783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e746e6c0-4127-42e7-b184-d8e90274bf4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.878941 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.878818    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e746e6c0-4127-42e7-b184-d8e90274bf4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.878941 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.878840    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e746e6c0-4127-42e7-b184-d8e90274bf4d-manifests\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.879226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.879209    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e746e6c0-4127-42e7-b184-d8e90274bf4d-manifests\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:46.881052 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:46.881029    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e746e6c0-4127-42e7-b184-d8e90274bf4d-tmp\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:47.724671 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.724645    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-72j5p\""
Sep 16 12:46:47.788118 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.788092    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:46:47.791376 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.791351    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e746e6c0-4127-42e7-b184-d8e90274bf4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:47.791515 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.791495    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e746e6c0-4127-42e7-b184-d8e90274bf4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:47.826313 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.826288    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:46:47.829462 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.829446    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:46:47.924822 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.924797    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:46:47.929223 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:47.929202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7r8w\" (UniqueName: \"kubernetes.io/projected/e746e6c0-4127-42e7-b184-d8e90274bf4d-kube-api-access-v7r8w\") pod \"opendatahub-operator-controller-manager-5c7bc95455-k7b8l\" (UID: \"e746e6c0-4127-42e7-b184-d8e90274bf4d\") " pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:48.145090 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:48.145004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:48.420452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:48.420127    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"]
Sep 16 12:46:48.425175 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:46:48.425143    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode746e6c0_4127_42e7_b184_d8e90274bf4d.slice/crio-aa1e22f69767693c78c592b3fcdf5933015332da9428d9d3f9081b743b54493b WatchSource:0}: Error finding container aa1e22f69767693c78c592b3fcdf5933015332da9428d9d3f9081b743b54493b: Status 404 returned error can't find the container with id aa1e22f69767693c78c592b3fcdf5933015332da9428d9d3f9081b743b54493b
Sep 16 12:46:48.506171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:48.506129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l" event={"ID":"e746e6c0-4127-42e7-b184-d8e90274bf4d","Type":"ContainerStarted","Data":"aa1e22f69767693c78c592b3fcdf5933015332da9428d9d3f9081b743b54493b"}
Sep 16 12:46:52.484835 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:52.484804    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-575fbb86f4-cndrn"
Sep 16 12:46:52.527203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:52.527171    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l" event={"ID":"e746e6c0-4127-42e7-b184-d8e90274bf4d","Type":"ContainerStarted","Data":"03cec2b85ffeec42134ba1547de6ed22b67ed0a5090f16b90ac83ef22395ffd7"}
Sep 16 12:46:53.532852 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:53.532818    2559 generic.go:356] "Generic (PLEG): container finished" podID="e746e6c0-4127-42e7-b184-d8e90274bf4d" containerID="03cec2b85ffeec42134ba1547de6ed22b67ed0a5090f16b90ac83ef22395ffd7" exitCode=0
Sep 16 12:46:53.533226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:53.532900    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l" event={"ID":"e746e6c0-4127-42e7-b184-d8e90274bf4d","Type":"ContainerDied","Data":"03cec2b85ffeec42134ba1547de6ed22b67ed0a5090f16b90ac83ef22395ffd7"}
Sep 16 12:46:54.538988 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:54.538949    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l" event={"ID":"e746e6c0-4127-42e7-b184-d8e90274bf4d","Type":"ContainerStarted","Data":"dca5c3f3e7de7e7454271b61971615f2c40f0d1517da1d81131c635d02ec88a9"}
Sep 16 12:46:54.539403 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:54.539039    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:46:54.563658 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:46:54.563613    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l" podStartSLOduration=4.050227374 podStartE2EDuration="8.563597644s" totalImagesPullingTime="3.492470464s" totalInitContainerRuntime="1.020899806s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:46:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:46:54.562529567 +0000 UTC m=+516.645972605" watchObservedRunningTime="2026-09-16 12:46:54.563597644 +0000 UTC m=+516.647040657"
Sep 16 12:47:05.545951 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:05.545919    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c7bc95455-k7b8l"
Sep 16 12:47:18.516395 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.516297    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"]
Sep 16 12:47:18.522151 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.522129    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.525255 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.525231    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 12:47:18.525540 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.525523    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 12:47:18.525638 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.525557    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:47:18.526047 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.526029    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kc96p\""
Sep 16 12:47:18.526132 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.526034    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:47:18.532013 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.531992    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"]
Sep 16 12:47:18.642085 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.642057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tmp\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.642271 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.642140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tls-certs\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.642271 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.642173    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5c8\" (UniqueName: \"kubernetes.io/projected/b3ea20f8-efcc-4b1d-9066-e42907f636e2-kube-api-access-9r5c8\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.743103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.743069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tls-certs\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.743269 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.743117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5c8\" (UniqueName: \"kubernetes.io/projected/b3ea20f8-efcc-4b1d-9066-e42907f636e2-kube-api-access-9r5c8\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.743269 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.743167    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tmp\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.743269 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:47:18.743213    2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 12:47:18.743407 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:47:18.743286    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tls-certs podName:b3ea20f8-efcc-4b1d-9066-e42907f636e2 nodeName:}" failed. No retries permitted until 2026-09-16 12:47:19.243270448 +0000 UTC m=+541.326713440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tls-certs") pod "kube-auth-proxy-8568fcf948-77q2n" (UID: "b3ea20f8-efcc-4b1d-9066-e42907f636e2") : secret "kube-auth-proxy-tls" not found
Sep 16 12:47:18.745435 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.745415    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tmp\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:18.753625 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:18.753602    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5c8\" (UniqueName: \"kubernetes.io/projected/b3ea20f8-efcc-4b1d-9066-e42907f636e2-kube-api-access-9r5c8\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:19.248332 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:19.248287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tls-certs\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:19.250741 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:19.250724    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3ea20f8-efcc-4b1d-9066-e42907f636e2-tls-certs\") pod \"kube-auth-proxy-8568fcf948-77q2n\" (UID: \"b3ea20f8-efcc-4b1d-9066-e42907f636e2\") " pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:19.433367 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:19.433310    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"
Sep 16 12:47:19.716963 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:19.716918    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8568fcf948-77q2n"]
Sep 16 12:47:19.719475 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:47:19.719453    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ea20f8_efcc_4b1d_9066_e42907f636e2.slice/crio-47256af6e6b618bcadf8a91c77d1a7afe8cab883fc91532a60263eea03fe1650 WatchSource:0}: Error finding container 47256af6e6b618bcadf8a91c77d1a7afe8cab883fc91532a60263eea03fe1650: Status 404 returned error can't find the container with id 47256af6e6b618bcadf8a91c77d1a7afe8cab883fc91532a60263eea03fe1650
Sep 16 12:47:20.654845 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:20.654800    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n" event={"ID":"b3ea20f8-efcc-4b1d-9066-e42907f636e2","Type":"ContainerStarted","Data":"47256af6e6b618bcadf8a91c77d1a7afe8cab883fc91532a60263eea03fe1650"}
Sep 16 12:47:22.665753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:22.665718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n" event={"ID":"b3ea20f8-efcc-4b1d-9066-e42907f636e2","Type":"ContainerStarted","Data":"e7a3ef252d7596808c0e9fd02f8db9e3a8258a1c127561475ace96967b263a18"}
Sep 16 12:47:22.684132 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:47:22.684053    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8568fcf948-77q2n" podStartSLOduration=1.9020971709999999 podStartE2EDuration="4.684034948s" totalImagesPullingTime="2.781937777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:22.683480977 +0000 UTC m=+544.766923991" watchObservedRunningTime="2026-09-16 12:47:22.684034948 +0000 UTC m=+544.767477962"
Sep 16 12:48:51.564443 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.564363    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"]
Sep 16 12:48:51.567700 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.567684    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.572385 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.572350    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 12:48:51.572516 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.572396    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:48:51.572516 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.572460    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:48:51.573031 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.573015    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 12:48:51.573196 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.573178    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gpmpt\""
Sep 16 12:48:51.580492 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.580468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"]
Sep 16 12:48:51.637541 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.637508    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd517a2-175f-4f07-a62b-d5733ceca442-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.637733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.637628    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bd517a2-175f-4f07-a62b-d5733ceca442-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.637733 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.637671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm86p\" (UniqueName: \"kubernetes.io/projected/7bd517a2-175f-4f07-a62b-d5733ceca442-kube-api-access-lm86p\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.738830 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.738789    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bd517a2-175f-4f07-a62b-d5733ceca442-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.738830 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.738846    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm86p\" (UniqueName: \"kubernetes.io/projected/7bd517a2-175f-4f07-a62b-d5733ceca442-kube-api-access-lm86p\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.739063 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.738869    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd517a2-175f-4f07-a62b-d5733ceca442-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.739063 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:48:51.739006    2559 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 16 12:48:51.739129 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:48:51.739066    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bd517a2-175f-4f07-a62b-d5733ceca442-plugin-serving-cert podName:7bd517a2-175f-4f07-a62b-d5733ceca442 nodeName:}" failed. No retries permitted until 2026-09-16 12:48:52.239048025 +0000 UTC m=+634.322491022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7bd517a2-175f-4f07-a62b-d5733ceca442-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-8vkn5" (UID: "7bd517a2-175f-4f07-a62b-d5733ceca442") : secret "plugin-serving-cert" not found
Sep 16 12:48:51.739573 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.739553    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7bd517a2-175f-4f07-a62b-d5733ceca442-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:51.750147 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:51.750121    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm86p\" (UniqueName: \"kubernetes.io/projected/7bd517a2-175f-4f07-a62b-d5733ceca442-kube-api-access-lm86p\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:52.243582 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:52.243539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd517a2-175f-4f07-a62b-d5733ceca442-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:52.246143 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:52.246103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7bd517a2-175f-4f07-a62b-d5733ceca442-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8vkn5\" (UID: \"7bd517a2-175f-4f07-a62b-d5733ceca442\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:52.491758 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:52.491718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"
Sep 16 12:48:52.784759 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:52.784731    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5"]
Sep 16 12:48:52.786739 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:48:52.786707    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd517a2_175f_4f07_a62b_d5733ceca442.slice/crio-e94d53d811c25731b97c10d4abca4f13691c74fcc6068a0001a12db53d892886 WatchSource:0}: Error finding container e94d53d811c25731b97c10d4abca4f13691c74fcc6068a0001a12db53d892886: Status 404 returned error can't find the container with id e94d53d811c25731b97c10d4abca4f13691c74fcc6068a0001a12db53d892886
Sep 16 12:48:53.047840 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:53.047753    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5" event={"ID":"7bd517a2-175f-4f07-a62b-d5733ceca442","Type":"ContainerStarted","Data":"e94d53d811c25731b97c10d4abca4f13691c74fcc6068a0001a12db53d892886"}
Sep 16 12:48:58.073669 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:58.073634    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5" event={"ID":"7bd517a2-175f-4f07-a62b-d5733ceca442","Type":"ContainerStarted","Data":"86a4ae5ba7a1b9233a299767cb6a1f9b99f44ade7e92ae1ca66095452bf77d39"}
Sep 16 12:48:58.092501 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:48:58.092446    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8vkn5" podStartSLOduration=1.938837283 podStartE2EDuration="7.092429788s" totalImagesPullingTime="5.153592505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:58.091242495 +0000 UTC m=+640.174685520" watchObservedRunningTime="2026-09-16 12:48:58.092429788 +0000 UTC m=+640.175872801"
Sep 16 12:49:01.864428 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:01.864387    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"]
Sep 16 12:49:01.868264 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:01.868240    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:01.870877 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:01.870852    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hsr4g\""
Sep 16 12:49:01.879954 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:01.879927    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"]
Sep 16 12:49:01.934897 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:01.934849    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghld\" (UniqueName: \"kubernetes.io/projected/7a84f3d8-d212-4476-8e40-031564c95273-kube-api-access-2ghld\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:01.934897 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:01.934905    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a84f3d8-d212-4476-8e40-031564c95273-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:02.036217 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.036177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghld\" (UniqueName: \"kubernetes.io/projected/7a84f3d8-d212-4476-8e40-031564c95273-kube-api-access-2ghld\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:02.036414 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.036227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a84f3d8-d212-4476-8e40-031564c95273-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:02.036625 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.036608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a84f3d8-d212-4476-8e40-031564c95273-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:02.045626 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.045594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghld\" (UniqueName: \"kubernetes.io/projected/7a84f3d8-d212-4476-8e40-031564c95273-kube-api-access-2ghld\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:02.179062 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.179022    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:02.458419 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.458390    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"]
Sep 16 12:49:02.461257 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:49:02.461227    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a84f3d8_d212_4476_8e40_031564c95273.slice/crio-21de8d70187b96ba5a717c0fb3b5be6d097210f31bddcfdc9cb9eb3c18e4e4d3 WatchSource:0}: Error finding container 21de8d70187b96ba5a717c0fb3b5be6d097210f31bddcfdc9cb9eb3c18e4e4d3: Status 404 returned error can't find the container with id 21de8d70187b96ba5a717c0fb3b5be6d097210f31bddcfdc9cb9eb3c18e4e4d3
Sep 16 12:49:02.993234 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:02.993153    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"]
Sep 16 12:49:03.000261 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:03.000183    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"]
Sep 16 12:49:07.127487 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.127420    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd" podUID="7a84f3d8-d212-4476-8e40-031564c95273" containerName="manager" containerID="cri-o://cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf" gracePeriod=2
Sep 16 12:49:07.366950 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.366926    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:07.369150 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:49:07.369121    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="7a84f3d8-d212-4476-8e40-031564c95273" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:07.489301 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.489257    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ghld\" (UniqueName: \"kubernetes.io/projected/7a84f3d8-d212-4476-8e40-031564c95273-kube-api-access-2ghld\") pod \"7a84f3d8-d212-4476-8e40-031564c95273\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") "
Sep 16 12:49:07.489301 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.489312    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a84f3d8-d212-4476-8e40-031564c95273-extensions-socket-volume\") pod \"7a84f3d8-d212-4476-8e40-031564c95273\" (UID: \"7a84f3d8-d212-4476-8e40-031564c95273\") "
Sep 16 12:49:07.489669 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.489635    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a84f3d8-d212-4476-8e40-031564c95273-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7a84f3d8-d212-4476-8e40-031564c95273" (UID: "7a84f3d8-d212-4476-8e40-031564c95273"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:49:07.491829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.491809    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a84f3d8-d212-4476-8e40-031564c95273-kube-api-access-2ghld" (OuterVolumeSpecName: "kube-api-access-2ghld") pod "7a84f3d8-d212-4476-8e40-031564c95273" (UID: "7a84f3d8-d212-4476-8e40-031564c95273"). InnerVolumeSpecName "kube-api-access-2ghld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:49:07.590967 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.590928    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghld\" (UniqueName: \"kubernetes.io/projected/7a84f3d8-d212-4476-8e40-031564c95273-kube-api-access-2ghld\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:49:07.590967 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:07.590960    2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a84f3d8-d212-4476-8e40-031564c95273-extensions-socket-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:49:08.133359 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:08.133299    2559 generic.go:356] "Generic (PLEG): container finished" podID="7a84f3d8-d212-4476-8e40-031564c95273" containerID="cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf" exitCode=2
Sep 16 12:49:08.133359 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:08.133359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:08.133812 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:08.133402    2559 scope.go:117] "RemoveContainer" containerID="cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf"
Sep 16 12:49:08.135637 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:49:08.135609    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="7a84f3d8-d212-4476-8e40-031564c95273" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:08.142545 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:08.142528    2559 scope.go:117] "RemoveContainer" containerID="cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf"
Sep 16 12:49:08.142849 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:49:08.142831    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf\": container with ID starting with cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf not found: ID does not exist" containerID="cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf"
Sep 16 12:49:08.142900 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:08.142859    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf"} err="failed to get container status \"cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf\": rpc error: code = NotFound desc = could not find container \"cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf\": container with ID starting with cfb2d757a05d5e73034411265aca4ff3d13b7e14c988cdac1d9680cf71905bdf not found: ID does not exist"
Sep 16 12:49:08.143958 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:49:08.143934    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="7a84f3d8-d212-4476-8e40-031564c95273" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:08.411021 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:49:08.410936    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-6fmgd\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="7a84f3d8-d212-4476-8e40-031564c95273" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-6fmgd"
Sep 16 12:49:08.412223 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:08.412201    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a84f3d8-d212-4476-8e40-031564c95273" path="/var/lib/kubelet/pods/7a84f3d8-d212-4476-8e40-031564c95273/volumes"
Sep 16 12:49:53.067281 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.067247    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:49:53.067789 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.067593    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a84f3d8-d212-4476-8e40-031564c95273" containerName="manager"
Sep 16 12:49:53.067789 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.067606    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a84f3d8-d212-4476-8e40-031564c95273" containerName="manager"
Sep 16 12:49:53.067789 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.067669    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a84f3d8-d212-4476-8e40-031564c95273" containerName="manager"
Sep 16 12:49:53.070629 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.070614    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.073153 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.073125    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 12:49:53.080541 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.080521    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:49:53.109946 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.109915    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:49:53.208083 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.208045    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7842f60e-a30c-497b-b11d-6841236c8dc0-config-file\") pod \"limitador-limitador-84994f7ffc-wxzvf\" (UID: \"7842f60e-a30c-497b-b11d-6841236c8dc0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.208083 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.208088    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsqv\" (UniqueName: \"kubernetes.io/projected/7842f60e-a30c-497b-b11d-6841236c8dc0-kube-api-access-prsqv\") pod \"limitador-limitador-84994f7ffc-wxzvf\" (UID: \"7842f60e-a30c-497b-b11d-6841236c8dc0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.308674 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.308628    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7842f60e-a30c-497b-b11d-6841236c8dc0-config-file\") pod \"limitador-limitador-84994f7ffc-wxzvf\" (UID: \"7842f60e-a30c-497b-b11d-6841236c8dc0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.308674 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.308679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsqv\" (UniqueName: \"kubernetes.io/projected/7842f60e-a30c-497b-b11d-6841236c8dc0-kube-api-access-prsqv\") pod \"limitador-limitador-84994f7ffc-wxzvf\" (UID: \"7842f60e-a30c-497b-b11d-6841236c8dc0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.309269 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.309249    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7842f60e-a30c-497b-b11d-6841236c8dc0-config-file\") pod \"limitador-limitador-84994f7ffc-wxzvf\" (UID: \"7842f60e-a30c-497b-b11d-6841236c8dc0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.316822 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.316790    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsqv\" (UniqueName: \"kubernetes.io/projected/7842f60e-a30c-497b-b11d-6841236c8dc0-kube-api-access-prsqv\") pod \"limitador-limitador-84994f7ffc-wxzvf\" (UID: \"7842f60e-a30c-497b-b11d-6841236c8dc0\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.381035 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.380948    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:53.657009 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.656954    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:49:53.659180 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:49:53.659150    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7842f60e_a30c_497b_b11d_6841236c8dc0.slice/crio-6249381d925f98eed25b63603d983a80060c95e3e06b536b4154ce61ef428cdb WatchSource:0}: Error finding container 6249381d925f98eed25b63603d983a80060c95e3e06b536b4154ce61ef428cdb: Status 404 returned error can't find the container with id 6249381d925f98eed25b63603d983a80060c95e3e06b536b4154ce61ef428cdb
Sep 16 12:49:53.694333 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.694301    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-vp4fm"]
Sep 16 12:49:53.698540 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.698524    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:49:53.700880 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.700860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9hx2j\""
Sep 16 12:49:53.705213 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.705170    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vp4fm"]
Sep 16 12:49:53.814610 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.814580    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvsmm\" (UniqueName: \"kubernetes.io/projected/6b827a52-25bc-4362-baa3-16b0948fb134-kube-api-access-lvsmm\") pod \"authorino-59c974bd48-vp4fm\" (UID: \"6b827a52-25bc-4362-baa3-16b0948fb134\") " pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:49:53.915464 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.915434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvsmm\" (UniqueName: \"kubernetes.io/projected/6b827a52-25bc-4362-baa3-16b0948fb134-kube-api-access-lvsmm\") pod \"authorino-59c974bd48-vp4fm\" (UID: \"6b827a52-25bc-4362-baa3-16b0948fb134\") " pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:49:53.925172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:53.925145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvsmm\" (UniqueName: \"kubernetes.io/projected/6b827a52-25bc-4362-baa3-16b0948fb134-kube-api-access-lvsmm\") pod \"authorino-59c974bd48-vp4fm\" (UID: \"6b827a52-25bc-4362-baa3-16b0948fb134\") " pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:49:54.009008 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:54.008981    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:49:54.277646 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:54.277529    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vp4fm"]
Sep 16 12:49:54.347763 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:54.347730    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf" event={"ID":"7842f60e-a30c-497b-b11d-6841236c8dc0","Type":"ContainerStarted","Data":"6249381d925f98eed25b63603d983a80060c95e3e06b536b4154ce61ef428cdb"}
Sep 16 12:49:54.446703 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:49:54.446671    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b827a52_25bc_4362_baa3_16b0948fb134.slice/crio-4de92221117ed38690587a1435e10d315120359ef6833c461963ea397ac6774b WatchSource:0}: Error finding container 4de92221117ed38690587a1435e10d315120359ef6833c461963ea397ac6774b: Status 404 returned error can't find the container with id 4de92221117ed38690587a1435e10d315120359ef6833c461963ea397ac6774b
Sep 16 12:49:55.354927 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:55.354877    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf" event={"ID":"7842f60e-a30c-497b-b11d-6841236c8dc0","Type":"ContainerStarted","Data":"c710d0eee2052e17ed3abccb41a8144689c8fbf8a4ddef39c449b3d1dd44aacc"}
Sep 16 12:49:55.355430 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:55.354989    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:49:55.356388 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:55.356361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vp4fm" event={"ID":"6b827a52-25bc-4362-baa3-16b0948fb134","Type":"ContainerStarted","Data":"4de92221117ed38690587a1435e10d315120359ef6833c461963ea397ac6774b"}
Sep 16 12:49:55.375411 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:55.375313    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf" podStartSLOduration=1.533249404 podStartE2EDuration="2.375290692s" totalImagesPullingTime="842.041288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:55.374879069 +0000 UTC m=+697.458322104" watchObservedRunningTime="2026-09-16 12:49:55.375290692 +0000 UTC m=+697.458733707"
Sep 16 12:49:56.361708 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:56.361665    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vp4fm" event={"ID":"6b827a52-25bc-4362-baa3-16b0948fb134","Type":"ContainerStarted","Data":"4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32"}
Sep 16 12:49:56.382481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:49:56.382430    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-vp4fm" podStartSLOduration=1.948634113 podStartE2EDuration="3.382412914s" totalImagesPullingTime="1.433778801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:49:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:49:56.380004636 +0000 UTC m=+698.463447649" watchObservedRunningTime="2026-09-16 12:49:56.382412914 +0000 UTC m=+698.465855976"
Sep 16 12:50:06.362936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:06.362906    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"
Sep 16 12:50:25.208974 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.208890    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vp4fm"]
Sep 16 12:50:25.209448 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.209113    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-vp4fm" podUID="6b827a52-25bc-4362-baa3-16b0948fb134" containerName="authorino" containerID="cri-o://4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32" gracePeriod=30
Sep 16 12:50:25.458016 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.457988    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:50:25.491049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.490953    2559 generic.go:356] "Generic (PLEG): container finished" podID="6b827a52-25bc-4362-baa3-16b0948fb134" containerID="4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32" exitCode=0
Sep 16 12:50:25.491049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.490996    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vp4fm"
Sep 16 12:50:25.491049 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.491043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vp4fm" event={"ID":"6b827a52-25bc-4362-baa3-16b0948fb134","Type":"ContainerDied","Data":"4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32"}
Sep 16 12:50:25.491332 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.491084    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vp4fm" event={"ID":"6b827a52-25bc-4362-baa3-16b0948fb134","Type":"ContainerDied","Data":"4de92221117ed38690587a1435e10d315120359ef6833c461963ea397ac6774b"}
Sep 16 12:50:25.491332 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.491102    2559 scope.go:117] "RemoveContainer" containerID="4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32"
Sep 16 12:50:25.499670 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.499647    2559 scope.go:117] "RemoveContainer" containerID="4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32"
Sep 16 12:50:25.499979 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:50:25.499959    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32\": container with ID starting with 4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32 not found: ID does not exist" containerID="4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32"
Sep 16 12:50:25.500048 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.499988    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32"} err="failed to get container status \"4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32\": rpc error: code = NotFound desc = could not find container \"4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32\": container with ID starting with 4009f637bc46377f8a08d5fddbb67f7dc58e52a78d449892be77fe87de161b32 not found: ID does not exist"
Sep 16 12:50:25.579535 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.579501    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvsmm\" (UniqueName: \"kubernetes.io/projected/6b827a52-25bc-4362-baa3-16b0948fb134-kube-api-access-lvsmm\") pod \"6b827a52-25bc-4362-baa3-16b0948fb134\" (UID: \"6b827a52-25bc-4362-baa3-16b0948fb134\") "
Sep 16 12:50:25.581763 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.581734    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b827a52-25bc-4362-baa3-16b0948fb134-kube-api-access-lvsmm" (OuterVolumeSpecName: "kube-api-access-lvsmm") pod "6b827a52-25bc-4362-baa3-16b0948fb134" (UID: "6b827a52-25bc-4362-baa3-16b0948fb134"). InnerVolumeSpecName "kube-api-access-lvsmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:50:25.680980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.680945    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvsmm\" (UniqueName: \"kubernetes.io/projected/6b827a52-25bc-4362-baa3-16b0948fb134-kube-api-access-lvsmm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:50:25.818062 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.818029    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vp4fm"]
Sep 16 12:50:25.830241 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:25.830209    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vp4fm"]
Sep 16 12:50:26.412423 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:50:26.412389    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b827a52-25bc-4362-baa3-16b0948fb134" path="/var/lib/kubelet/pods/6b827a52-25bc-4362-baa3-16b0948fb134/volumes"
Sep 16 12:51:08.411481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:08.411447    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:51:30.551790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.551760    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"]
Sep 16 12:51:30.552288 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.552274    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b827a52-25bc-4362-baa3-16b0948fb134" containerName="authorino"
Sep 16 12:51:30.552360 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.552291    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b827a52-25bc-4362-baa3-16b0948fb134" containerName="authorino"
Sep 16 12:51:30.552416 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.552403    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b827a52-25bc-4362-baa3-16b0948fb134" containerName="authorino"
Sep 16 12:51:30.556702 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.556681    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.559133 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.559112    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:51:30.559897 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.559879    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 12:51:30.560008 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.559892    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:51:30.560008 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.559882    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q4h2f\""
Sep 16 12:51:30.565547 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.565528    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"]
Sep 16 12:51:30.638770 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.638727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf39c4-ec29-4273-ab54-62522f131a6e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.638770 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.638772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.639026 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.638854    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnwk\" (UniqueName: \"kubernetes.io/projected/f8bf39c4-ec29-4273-ab54-62522f131a6e-kube-api-access-fnnwk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.639026 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.638923    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.639026 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.638986    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.639135 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.639072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.739787 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.739738    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnwk\" (UniqueName: \"kubernetes.io/projected/f8bf39c4-ec29-4273-ab54-62522f131a6e-kube-api-access-fnnwk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.739909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.739802    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.739909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.739839    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.739909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.739874    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.739909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.739895    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf39c4-ec29-4273-ab54-62522f131a6e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.740107 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.739912    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.740279 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.740257    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.740375 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.740346    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.740420 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.740381    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.742273 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.742248    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8bf39c4-ec29-4273-ab54-62522f131a6e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.742606 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.742591    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8bf39c4-ec29-4273-ab54-62522f131a6e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.747836 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.747811    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnwk\" (UniqueName: \"kubernetes.io/projected/f8bf39c4-ec29-4273-ab54-62522f131a6e-kube-api-access-fnnwk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w\" (UID: \"f8bf39c4-ec29-4273-ab54-62522f131a6e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.868239 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.868162    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:30.957541 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:30.957508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:51:31.143792 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:31.143693    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"]
Sep 16 12:51:31.146778 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:51:31.146745    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bf39c4_ec29_4273_ab54_62522f131a6e.slice/crio-844186fead50b82098a1e54ba728c8b2877352c9d096f411a96f7e7c1a70c2de WatchSource:0}: Error finding container 844186fead50b82098a1e54ba728c8b2877352c9d096f411a96f7e7c1a70c2de: Status 404 returned error can't find the container with id 844186fead50b82098a1e54ba728c8b2877352c9d096f411a96f7e7c1a70c2de
Sep 16 12:51:31.785652 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:31.785616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w" event={"ID":"f8bf39c4-ec29-4273-ab54-62522f131a6e","Type":"ContainerStarted","Data":"844186fead50b82098a1e54ba728c8b2877352c9d096f411a96f7e7c1a70c2de"}
Sep 16 12:51:32.791983 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:32.791947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w" event={"ID":"f8bf39c4-ec29-4273-ab54-62522f131a6e","Type":"ContainerStarted","Data":"0338cbaf973f4427b71ad664a8be0c0ae35e09c45b46dc88169f799e782608ba"}
Sep 16 12:51:32.809864 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:32.809815    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w" podStartSLOduration=1.644599792 podStartE2EDuration="2.809799212s" totalImagesPullingTime="1.16519942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:32.80858941 +0000 UTC m=+794.892032435" watchObservedRunningTime="2026-09-16 12:51:32.809799212 +0000 UTC m=+794.893242225"
Sep 16 12:51:35.251803 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.251761    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"]
Sep 16 12:51:35.255251 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.255229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.257813 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.257793    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 16 12:51:35.264861 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.264839    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"]
Sep 16 12:51:35.384875 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.384838    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.384875 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.384878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542b8a7d-4fe4-4818-a18c-8c8fd092f562-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.385127 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.384903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf58w\" (UniqueName: \"kubernetes.io/projected/542b8a7d-4fe4-4818-a18c-8c8fd092f562-kube-api-access-tf58w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.385127 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.385018    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.385127 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.385040    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.385127 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.385116    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.486501 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.486462    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.486693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.486518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.486693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.486533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542b8a7d-4fe4-4818-a18c-8c8fd092f562-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.486693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.486550    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf58w\" (UniqueName: \"kubernetes.io/projected/542b8a7d-4fe4-4818-a18c-8c8fd092f562-kube-api-access-tf58w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.486693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.486631    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.486693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.486645    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.487026 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.487001    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.487091 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.487035    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.487156 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.487093    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.489043 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.489023    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542b8a7d-4fe4-4818-a18c-8c8fd092f562-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.489348 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.489296    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542b8a7d-4fe4-4818-a18c-8c8fd092f562-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.495607 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.495586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf58w\" (UniqueName: \"kubernetes.io/projected/542b8a7d-4fe4-4818-a18c-8c8fd092f562-kube-api-access-tf58w\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vrmzp\" (UID: \"542b8a7d-4fe4-4818-a18c-8c8fd092f562\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.565968 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.565873    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:35.840226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:35.840173    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"]
Sep 16 12:51:35.842655 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:51:35.842619    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542b8a7d_4fe4_4818_a18c_8c8fd092f562.slice/crio-749a77971ba2f0addd543643a2c37c85e70447fa25edbeb3f8752daf88bb8ea9 WatchSource:0}: Error finding container 749a77971ba2f0addd543643a2c37c85e70447fa25edbeb3f8752daf88bb8ea9: Status 404 returned error can't find the container with id 749a77971ba2f0addd543643a2c37c85e70447fa25edbeb3f8752daf88bb8ea9
Sep 16 12:51:36.155349 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:36.155256    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:51:36.810303 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:36.810258    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp" event={"ID":"542b8a7d-4fe4-4818-a18c-8c8fd092f562","Type":"ContainerStarted","Data":"a7b4454a62c351fcd0d41a9e26c61056a0767d734509c9a563c4e8d9f8f63d53"}
Sep 16 12:51:36.810772 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:36.810315    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp" event={"ID":"542b8a7d-4fe4-4818-a18c-8c8fd092f562","Type":"ContainerStarted","Data":"749a77971ba2f0addd543643a2c37c85e70447fa25edbeb3f8752daf88bb8ea9"}
Sep 16 12:51:36.831768 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:36.831719    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp" podStartSLOduration=1.6450413369999999 podStartE2EDuration="1.831704863s" totalImagesPullingTime="186.663526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:51:36.830345032 +0000 UTC m=+798.913788059" watchObservedRunningTime="2026-09-16 12:51:36.831704863 +0000 UTC m=+798.915147877"
Sep 16 12:51:40.868418 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:40.868339    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:40.868418 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:40.868406    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:40.881243 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:40.881216    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:41.846452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:41.846425    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w"
Sep 16 12:51:45.566920 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:45.566888    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:45.567284 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:45.567168    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:45.579812 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:45.579789    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:45.865668 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:45.865585    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vrmzp"
Sep 16 12:51:53.263160 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:51:53.263114    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:52:12.257305 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:12.257265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:52:23.055372 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:23.055329    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:52:40.953099 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:40.953065    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:52:47.663841 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.663810    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"]
Sep 16 12:52:47.667113 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.667096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.669431 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.669410    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 16 12:52:47.677096 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.677075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"]
Sep 16 12:52:47.720171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.720136    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.720368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.720233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.720368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.720253    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.720368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.720274    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzcjv\" (UniqueName: \"kubernetes.io/projected/b0a2f058-2786-422b-9410-1e0b517a366e-kube-api-access-wzcjv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.720368 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.720300    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a2f058-2786-422b-9410-1e0b517a366e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.720532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.720389    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.821866 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.821827    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822063 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.821881    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822063 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.821946    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822063 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.821967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822243 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.822107    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzcjv\" (UniqueName: \"kubernetes.io/projected/b0a2f058-2786-422b-9410-1e0b517a366e-kube-api-access-wzcjv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822243 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.822155    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a2f058-2786-422b-9410-1e0b517a366e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822376 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.822269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822458 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.822434    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.822597 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.822569    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.824433 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.824412    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0a2f058-2786-422b-9410-1e0b517a366e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.824696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.824675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0a2f058-2786-422b-9410-1e0b517a366e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.834791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.834770    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzcjv\" (UniqueName: \"kubernetes.io/projected/b0a2f058-2786-422b-9410-1e0b517a366e-kube-api-access-wzcjv\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-65flk\" (UID: \"b0a2f058-2786-422b-9410-1e0b517a366e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:47.977111 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:47.977025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:48.248088 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:48.248063    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"]
Sep 16 12:52:48.250883 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:52:48.250856    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0a2f058_2786_422b_9410_1e0b517a366e.slice/crio-0e5e942c24cb1dffc7fa2cf9453c97efe63046f6e2bcdf7dc21e2b5a6c01490d WatchSource:0}: Error finding container 0e5e942c24cb1dffc7fa2cf9453c97efe63046f6e2bcdf7dc21e2b5a6c01490d: Status 404 returned error can't find the container with id 0e5e942c24cb1dffc7fa2cf9453c97efe63046f6e2bcdf7dc21e2b5a6c01490d
Sep 16 12:52:48.252113 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:48.252097    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:52:49.140565 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:49.140525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk" event={"ID":"b0a2f058-2786-422b-9410-1e0b517a366e","Type":"ContainerStarted","Data":"50aa266ec585ce8b4973db7cdf012c165815977b418d3f82b538c9521ca95196"}
Sep 16 12:52:49.140958 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:49.140572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk" event={"ID":"b0a2f058-2786-422b-9410-1e0b517a366e","Type":"ContainerStarted","Data":"0e5e942c24cb1dffc7fa2cf9453c97efe63046f6e2bcdf7dc21e2b5a6c01490d"}
Sep 16 12:52:49.161082 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:49.161002    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk" podStartSLOduration=1.9457611620000002 podStartE2EDuration="2.160974339s" totalImagesPullingTime="215.213177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:52:49.159836033 +0000 UTC m=+871.243279048" watchObservedRunningTime="2026-09-16 12:52:49.160974339 +0000 UTC m=+871.244417355"
Sep 16 12:52:49.956340 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:49.956290    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:52:57.977661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:57.977628    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:57.978157 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:57.977687    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:57.989744 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:57.989715    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:52:58.190583 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:52:58.190543    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-65flk"
Sep 16 12:54:11.304844 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.304810    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cb5d9656f-dxhbx"]
Sep 16 12:54:11.308026 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.308009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.311203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.311181    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9hx2j\""
Sep 16 12:54:11.311306 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.311230    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 12:54:11.316007 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.315990    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb5d9656f-dxhbx"]
Sep 16 12:54:11.369181 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.369154    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdltl\" (UniqueName: \"kubernetes.io/projected/deb60e74-887e-46af-bc89-3216fd090dc7-kube-api-access-cdltl\") pod \"authorino-7cb5d9656f-dxhbx\" (UID: \"deb60e74-887e-46af-bc89-3216fd090dc7\") " pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.369369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.369189    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/deb60e74-887e-46af-bc89-3216fd090dc7-tls-cert\") pod \"authorino-7cb5d9656f-dxhbx\" (UID: \"deb60e74-887e-46af-bc89-3216fd090dc7\") " pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.469698 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.469661    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdltl\" (UniqueName: \"kubernetes.io/projected/deb60e74-887e-46af-bc89-3216fd090dc7-kube-api-access-cdltl\") pod \"authorino-7cb5d9656f-dxhbx\" (UID: \"deb60e74-887e-46af-bc89-3216fd090dc7\") " pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.469698 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.469701    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/deb60e74-887e-46af-bc89-3216fd090dc7-tls-cert\") pod \"authorino-7cb5d9656f-dxhbx\" (UID: \"deb60e74-887e-46af-bc89-3216fd090dc7\") " pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.472268 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.472249    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/deb60e74-887e-46af-bc89-3216fd090dc7-tls-cert\") pod \"authorino-7cb5d9656f-dxhbx\" (UID: \"deb60e74-887e-46af-bc89-3216fd090dc7\") " pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.479551 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.479524    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdltl\" (UniqueName: \"kubernetes.io/projected/deb60e74-887e-46af-bc89-3216fd090dc7-kube-api-access-cdltl\") pod \"authorino-7cb5d9656f-dxhbx\" (UID: \"deb60e74-887e-46af-bc89-3216fd090dc7\") " pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.618357 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.618244    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cb5d9656f-dxhbx"
Sep 16 12:54:11.891355 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:11.891242    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cb5d9656f-dxhbx"]
Sep 16 12:54:11.893973 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:11.893948    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb60e74_887e_46af_bc89_3216fd090dc7.slice/crio-c5f14566e79bd24fce835fa610f1c7d9e726429b29d695c2e68a5871adeb4a39 WatchSource:0}: Error finding container c5f14566e79bd24fce835fa610f1c7d9e726429b29d695c2e68a5871adeb4a39: Status 404 returned error can't find the container with id c5f14566e79bd24fce835fa610f1c7d9e726429b29d695c2e68a5871adeb4a39
Sep 16 12:54:12.516489 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:12.516453    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb5d9656f-dxhbx" event={"ID":"deb60e74-887e-46af-bc89-3216fd090dc7","Type":"ContainerStarted","Data":"c5f14566e79bd24fce835fa610f1c7d9e726429b29d695c2e68a5871adeb4a39"}
Sep 16 12:54:14.528481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:14.528387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cb5d9656f-dxhbx" event={"ID":"deb60e74-887e-46af-bc89-3216fd090dc7","Type":"ContainerStarted","Data":"484ee78e02acdaf60e6a33c69fa293fe63074b82d15d705db1f3ec986d5521ad"}
Sep 16 12:54:14.554076 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:14.554011    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cb5d9656f-dxhbx" podStartSLOduration=1.252447568 podStartE2EDuration="3.553995499s" totalImagesPullingTime="2.301547931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:14.552648119 +0000 UTC m=+956.636091135" watchObservedRunningTime="2026-09-16 12:54:14.553995499 +0000 UTC m=+956.637438507"
Sep 16 12:54:41.494884 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:41.494802    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:54:44.213513 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.213480    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-6398cb"]
Sep 16 12:54:44.216788 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.216770    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:44.219193 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.219176    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z9lzx\""
Sep 16 12:54:44.222607 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.222583    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-6398cb"]
Sep 16 12:54:44.367571 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.367534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbpwk\" (UniqueName: \"kubernetes.io/projected/b682cc29-8e01-4fbd-a1e9-01e19a045b26-kube-api-access-qbpwk\") pod \"test-curl-7553-6398cb\" (UID: \"b682cc29-8e01-4fbd-a1e9-01e19a045b26\") " pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:44.468506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.468409    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbpwk\" (UniqueName: \"kubernetes.io/projected/b682cc29-8e01-4fbd-a1e9-01e19a045b26-kube-api-access-qbpwk\") pod \"test-curl-7553-6398cb\" (UID: \"b682cc29-8e01-4fbd-a1e9-01e19a045b26\") " pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:44.476384 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.476357    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbpwk\" (UniqueName: \"kubernetes.io/projected/b682cc29-8e01-4fbd-a1e9-01e19a045b26-kube-api-access-qbpwk\") pod \"test-curl-7553-6398cb\" (UID: \"b682cc29-8e01-4fbd-a1e9-01e19a045b26\") " pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:44.527371 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.527308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:44.798852 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:44.798816    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-6398cb"]
Sep 16 12:54:44.801628 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:44.801600    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb682cc29_8e01_4fbd_a1e9_01e19a045b26.slice/crio-d2a0fa9603fdd48f02c6b521f0079fcd4729cb5a0b736e492319ad797e97ada2 WatchSource:0}: Error finding container d2a0fa9603fdd48f02c6b521f0079fcd4729cb5a0b736e492319ad797e97ada2: Status 404 returned error can't find the container with id d2a0fa9603fdd48f02c6b521f0079fcd4729cb5a0b736e492319ad797e97ada2
Sep 16 12:54:45.413680 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.413650    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"]
Sep 16 12:54:45.417289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.417262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.419654 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.419630    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-490797\""
Sep 16 12:54:45.419778 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.419713    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-490797-dockercfg-ltqcj\""
Sep 16 12:54:45.432849 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.432820    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"]
Sep 16 12:54:45.578254 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.578211    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2rfc\" (UniqueName: \"kubernetes.io/projected/fd5eba89-f4e7-41eb-810b-d7936c96740b-kube-api-access-d2rfc\") pod \"payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.578461 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.578384    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5eba89-f4e7-41eb-810b-d7936c96740b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.679067 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.678970    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5eba89-f4e7-41eb-810b-d7936c96740b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.679289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.679099    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2rfc\" (UniqueName: \"kubernetes.io/projected/fd5eba89-f4e7-41eb-810b-d7936c96740b-kube-api-access-d2rfc\") pod \"payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.679856 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.679787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5eba89-f4e7-41eb-810b-d7936c96740b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.679962 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.679862    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-6398cb" event={"ID":"b682cc29-8e01-4fbd-a1e9-01e19a045b26","Type":"ContainerStarted","Data":"d2a0fa9603fdd48f02c6b521f0079fcd4729cb5a0b736e492319ad797e97ada2"}
Sep 16 12:54:45.691346 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.691266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2rfc\" (UniqueName: \"kubernetes.io/projected/fd5eba89-f4e7-41eb-810b-d7936c96740b-kube-api-access-d2rfc\") pod \"payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:45.730201 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:45.730166    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:46.022838 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:46.022799    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"]
Sep 16 12:54:46.620513 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:46.620472    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5eba89_f4e7_41eb_810b_d7936c96740b.slice/crio-442f7167b693e46b991dac553c17885a425581cc043f4e77e4bf0ca242ad0543 WatchSource:0}: Error finding container 442f7167b693e46b991dac553c17885a425581cc043f4e77e4bf0ca242ad0543: Status 404 returned error can't find the container with id 442f7167b693e46b991dac553c17885a425581cc043f4e77e4bf0ca242ad0543
Sep 16 12:54:46.685760 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:46.685731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b" event={"ID":"fd5eba89-f4e7-41eb-810b-d7936c96740b","Type":"ContainerStarted","Data":"442f7167b693e46b991dac553c17885a425581cc043f4e77e4bf0ca242ad0543"}
Sep 16 12:54:47.500906 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.500870    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"]
Sep 16 12:54:47.504365 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.504284    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.508842 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.508820    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-586bab-dockercfg-zrl8z\""
Sep 16 12:54:47.509108 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.509092    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-586bab\""
Sep 16 12:54:47.521369 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.521342    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"]
Sep 16 12:54:47.598949 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.598907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8spw\" (UniqueName: \"kubernetes.io/projected/36fdc0d0-2fb0-4d96-90e3-09955e545f71-kube-api-access-m8spw\") pod \"payload-processing-e2e-inf-a-586bab-89b68857c-jtc45\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.599133 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.599014    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fdc0d0-2fb0-4d96-90e3-09955e545f71-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-586bab-89b68857c-jtc45\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.693583 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.693426    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-6398cb" event={"ID":"b682cc29-8e01-4fbd-a1e9-01e19a045b26","Type":"ContainerStarted","Data":"cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0"}
Sep 16 12:54:47.700753 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.700679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fdc0d0-2fb0-4d96-90e3-09955e545f71-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-586bab-89b68857c-jtc45\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.700909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.700872    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8spw\" (UniqueName: \"kubernetes.io/projected/36fdc0d0-2fb0-4d96-90e3-09955e545f71-kube-api-access-m8spw\") pod \"payload-processing-e2e-inf-a-586bab-89b68857c-jtc45\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.701602 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.701580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fdc0d0-2fb0-4d96-90e3-09955e545f71-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-586bab-89b68857c-jtc45\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.709561 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.709511    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-6398cb" podStartSLOduration=1.8345134810000001 podStartE2EDuration="3.709493379s" totalImagesPullingTime="1.874979898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:47.708092157 +0000 UTC m=+989.791535173" watchObservedRunningTime="2026-09-16 12:54:47.709493379 +0000 UTC m=+989.792936394"
Sep 16 12:54:47.712833 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.712807    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8spw\" (UniqueName: \"kubernetes.io/projected/36fdc0d0-2fb0-4d96-90e3-09955e545f71-kube-api-access-m8spw\") pod \"payload-processing-e2e-inf-a-586bab-89b68857c-jtc45\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") " pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:47.819151 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:47.819009    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:48.107253 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:48.107223    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"]
Sep 16 12:54:48.351953 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:48.351914    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-6398cb"]
Sep 16 12:54:48.357787 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:48.357681    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-6398cb"]
Sep 16 12:54:48.662236 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:48.662204    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fdc0d0_2fb0_4d96_90e3_09955e545f71.slice/crio-9e6767da2bb1042132952e6b5a0e0d396cf851acf72464437c010041063a56e1 WatchSource:0}: Error finding container 9e6767da2bb1042132952e6b5a0e0d396cf851acf72464437c010041063a56e1: Status 404 returned error can't find the container with id 9e6767da2bb1042132952e6b5a0e0d396cf851acf72464437c010041063a56e1
Sep 16 12:54:48.699585 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:48.699556    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45" event={"ID":"36fdc0d0-2fb0-4d96-90e3-09955e545f71","Type":"ContainerStarted","Data":"9e6767da2bb1042132952e6b5a0e0d396cf851acf72464437c010041063a56e1"}
Sep 16 12:54:48.699961 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:54:48.699742    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-6398cb" container="test-curl-7553-6398cb"
Sep 16 12:54:49.705926 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:49.705884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45" event={"ID":"36fdc0d0-2fb0-4d96-90e3-09955e545f71","Type":"ContainerStarted","Data":"88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679"}
Sep 16 12:54:49.706450 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:49.705958    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:54:49.707698 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:49.707669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b" event={"ID":"fd5eba89-f4e7-41eb-810b-d7936c96740b","Type":"ContainerStarted","Data":"8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914"}
Sep 16 12:54:49.707835 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:49.707743    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-6398cb" podUID="b682cc29-8e01-4fbd-a1e9-01e19a045b26" containerName="test-curl-7553-6398cb" containerID="cri-o://cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0" gracePeriod=2
Sep 16 12:54:49.728068 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:49.728016    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45" podStartSLOduration=2.329991959 podStartE2EDuration="2.727993757s" totalImagesPullingTime="398.001798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:49.725645495 +0000 UTC m=+991.809088534" watchObservedRunningTime="2026-09-16 12:54:49.727993757 +0000 UTC m=+991.811436772"
Sep 16 12:54:49.744896 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:49.744846    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b" podStartSLOduration=2.351976372 podStartE2EDuration="4.744829228s" totalImagesPullingTime="2.392852856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:49.743924972 +0000 UTC m=+991.827368008" watchObservedRunningTime="2026-09-16 12:54:49.744829228 +0000 UTC m=+991.828272242"
Sep 16 12:54:50.712217 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:50.712181    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:54:51.871463 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:51.871440    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:52.040686 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.040578    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbpwk\" (UniqueName: \"kubernetes.io/projected/b682cc29-8e01-4fbd-a1e9-01e19a045b26-kube-api-access-qbpwk\") pod \"b682cc29-8e01-4fbd-a1e9-01e19a045b26\" (UID: \"b682cc29-8e01-4fbd-a1e9-01e19a045b26\") "
Sep 16 12:54:52.042936 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.042910    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b682cc29-8e01-4fbd-a1e9-01e19a045b26-kube-api-access-qbpwk" (OuterVolumeSpecName: "kube-api-access-qbpwk") pod "b682cc29-8e01-4fbd-a1e9-01e19a045b26" (UID: "b682cc29-8e01-4fbd-a1e9-01e19a045b26"). InnerVolumeSpecName "kube-api-access-qbpwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:54:52.142225 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.142192    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbpwk\" (UniqueName: \"kubernetes.io/projected/b682cc29-8e01-4fbd-a1e9-01e19a045b26-kube-api-access-qbpwk\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:54:52.411445 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.411368    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b682cc29-8e01-4fbd-a1e9-01e19a045b26" path="/var/lib/kubelet/pods/b682cc29-8e01-4fbd-a1e9-01e19a045b26/volumes"
Sep 16 12:54:52.722559 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.722526    2559 generic.go:356] "Generic (PLEG): container finished" podID="b682cc29-8e01-4fbd-a1e9-01e19a045b26" containerID="cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0" exitCode=137
Sep 16 12:54:52.722748 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.722570    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-6398cb"
Sep 16 12:54:52.722748 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.722579    2559 scope.go:117] "RemoveContainer" containerID="cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0"
Sep 16 12:54:52.731567 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.731485    2559 scope.go:117] "RemoveContainer" containerID="cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0"
Sep 16 12:54:52.732880 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:54:52.732855    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0\": container with ID starting with cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0 not found: ID does not exist" containerID="cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0"
Sep 16 12:54:52.732986 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:52.732892    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0"} err="failed to get container status \"cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0\": rpc error: code = NotFound desc = could not find container \"cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0\": container with ID starting with cdb79a4aa5558f3a791fd4782b648f0c6a8b1f757faf9e490437f77d69288cc0 not found: ID does not exist"
Sep 16 12:54:55.514430 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.514402    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"]
Sep 16 12:54:55.514790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.514744    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b682cc29-8e01-4fbd-a1e9-01e19a045b26" containerName="test-curl-7553-6398cb"
Sep 16 12:54:55.514790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.514756    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b682cc29-8e01-4fbd-a1e9-01e19a045b26" containerName="test-curl-7553-6398cb"
Sep 16 12:54:55.514865 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.514828    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b682cc29-8e01-4fbd-a1e9-01e19a045b26" containerName="test-curl-7553-6398cb"
Sep 16 12:54:55.519481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.519464    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.521865 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.521851    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a24d8b\""
Sep 16 12:54:55.526600 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.526436    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"]
Sep 16 12:54:55.571808 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.571774    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"]
Sep 16 12:54:55.573293 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.573275    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b96a68e-1b91-4a69-adb1-f01b57002d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.573435 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.573336    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7rp\" (UniqueName: \"kubernetes.io/projected/7b96a68e-1b91-4a69-adb1-f01b57002d5d-kube-api-access-ql7rp\") pod \"payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.575193 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.575180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.577790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.577767    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a24d8b-dockercfg-nb9hc\""
Sep 16 12:54:55.589436 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.589413    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"]
Sep 16 12:54:55.674005 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.673978    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805418a0-de25-4fa3-ae83-434958cc0b5f-plugins-config-volume\") pod \"payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") " pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.674159 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.674016    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b96a68e-1b91-4a69-adb1-f01b57002d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.674159 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.674082    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7rp\" (UniqueName: \"kubernetes.io/projected/7b96a68e-1b91-4a69-adb1-f01b57002d5d-kube-api-access-ql7rp\") pod \"payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.674159 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.674128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgrj\" (UniqueName: \"kubernetes.io/projected/805418a0-de25-4fa3-ae83-434958cc0b5f-kube-api-access-gdgrj\") pod \"payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") " pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.674673 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.674657    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b96a68e-1b91-4a69-adb1-f01b57002d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.684682 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.684661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7rp\" (UniqueName: \"kubernetes.io/projected/7b96a68e-1b91-4a69-adb1-f01b57002d5d-kube-api-access-ql7rp\") pod \"payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.775624 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.775533    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805418a0-de25-4fa3-ae83-434958cc0b5f-plugins-config-volume\") pod \"payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") " pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.775624 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.775603    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgrj\" (UniqueName: \"kubernetes.io/projected/805418a0-de25-4fa3-ae83-434958cc0b5f-kube-api-access-gdgrj\") pod \"payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") " pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.776221 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.776201    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805418a0-de25-4fa3-ae83-434958cc0b5f-plugins-config-volume\") pod \"payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") " pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.784395 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.784366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgrj\" (UniqueName: \"kubernetes.io/projected/805418a0-de25-4fa3-ae83-434958cc0b5f-kube-api-access-gdgrj\") pod \"payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") " pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:55.830379 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.830343    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:55.885916 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:55.885627    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:56.108645 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.108572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"]
Sep 16 12:54:56.111699 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:56.111670    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b96a68e_1b91_4a69_adb1_f01b57002d5d.slice/crio-e292faebfeaf647afdc55be761195bc3cf2b5ac46323171b4854a7e9ccf37bbe WatchSource:0}: Error finding container e292faebfeaf647afdc55be761195bc3cf2b5ac46323171b4854a7e9ccf37bbe: Status 404 returned error can't find the container with id e292faebfeaf647afdc55be761195bc3cf2b5ac46323171b4854a7e9ccf37bbe
Sep 16 12:54:56.175155 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.175124    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"]
Sep 16 12:54:56.176840 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:56.176811    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805418a0_de25_4fa3_ae83_434958cc0b5f.slice/crio-33d67f9c03a2efcbd2ea8e129bc77a7cd732556d25fd0c92be85887cf5922ae9 WatchSource:0}: Error finding container 33d67f9c03a2efcbd2ea8e129bc77a7cd732556d25fd0c92be85887cf5922ae9: Status 404 returned error can't find the container with id 33d67f9c03a2efcbd2ea8e129bc77a7cd732556d25fd0c92be85887cf5922ae9
Sep 16 12:54:56.743990 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.743956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v" event={"ID":"7b96a68e-1b91-4a69-adb1-f01b57002d5d","Type":"ContainerStarted","Data":"2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37"}
Sep 16 12:54:56.744486 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.744020    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v" event={"ID":"7b96a68e-1b91-4a69-adb1-f01b57002d5d","Type":"ContainerStarted","Data":"e292faebfeaf647afdc55be761195bc3cf2b5ac46323171b4854a7e9ccf37bbe"}
Sep 16 12:54:56.744486 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.744050    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:54:56.745696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.745671    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7" event={"ID":"805418a0-de25-4fa3-ae83-434958cc0b5f","Type":"ContainerStarted","Data":"3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5"}
Sep 16 12:54:56.745799 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.745718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7" event={"ID":"805418a0-de25-4fa3-ae83-434958cc0b5f","Type":"ContainerStarted","Data":"33d67f9c03a2efcbd2ea8e129bc77a7cd732556d25fd0c92be85887cf5922ae9"}
Sep 16 12:54:56.745799 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.745754    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:54:56.762530 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.762481    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v" podStartSLOduration=1.76246174 podStartE2EDuration="1.76246174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:56.762089078 +0000 UTC m=+998.845532108" watchObservedRunningTime="2026-09-16 12:54:56.76246174 +0000 UTC m=+998.845904756"
Sep 16 12:54:56.786904 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.786861    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7" podStartSLOduration=1.786846368 podStartE2EDuration="1.786846368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:54:56.784756039 +0000 UTC m=+998.868199079" watchObservedRunningTime="2026-09-16 12:54:56.786846368 +0000 UTC m=+998.870289380"
Sep 16 12:54:56.835148 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.835117    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"]
Sep 16 12:54:56.839237 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.839215    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.842627 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.842605    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:54:56.842760 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.842682    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:54:56.842828 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.842756    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:54:56.843348 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.843312    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-79dbf9e9-serving-cert\""
Sep 16 12:54:56.843436 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.843406    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nm49v\""
Sep 16 12:54:56.853149 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.853126    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"]
Sep 16 12:54:56.886273 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.886236    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.886415 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.886374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-metrics-certs\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.886454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.886435    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5b035-4654-46a6-a37c-957aa9c5dfb5-openshift-service-ca\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.886508 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.886480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8slxc\" (UniqueName: \"kubernetes.io/projected/52e5b035-4654-46a6-a37c-957aa9c5dfb5-kube-api-access-8slxc\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.987680 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.987643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.987829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.987713    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-metrics-certs\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.987829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.987742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5b035-4654-46a6-a37c-957aa9c5dfb5-openshift-service-ca\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.987829 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.987769    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8slxc\" (UniqueName: \"kubernetes.io/projected/52e5b035-4654-46a6-a37c-957aa9c5dfb5-kube-api-access-8slxc\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.987972 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:54:56.987826    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-serving-cert: secret "maas-api-e2e-ait-79dbf9e9-serving-cert" not found
Sep 16 12:54:56.987972 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:54:56.987917    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls podName:52e5b035-4654-46a6-a37c-957aa9c5dfb5 nodeName:}" failed. No retries permitted until 2026-09-16 12:54:57.487896462 +0000 UTC m=+999.571339468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls") pod "maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" (UID: "52e5b035-4654-46a6-a37c-957aa9c5dfb5") : secret "maas-api-e2e-ait-79dbf9e9-serving-cert" not found
Sep 16 12:54:56.988541 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.988472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5b035-4654-46a6-a37c-957aa9c5dfb5-openshift-service-ca\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.990460 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.990437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-metrics-certs\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:56.999347 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:56.999265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8slxc\" (UniqueName: \"kubernetes.io/projected/52e5b035-4654-46a6-a37c-957aa9c5dfb5-kube-api-access-8slxc\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:57.492666 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:57.492205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:57.495800 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:57.495774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls\") pod \"maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:57.753497 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:57.753411    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:54:58.036298 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:58.036274    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"]
Sep 16 12:54:58.038815 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:54:58.038775    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e5b035_4654_46a6_a37c_957aa9c5dfb5.slice/crio-506dd8210ec1842c76ef3ff8e406daeb9b4e4356fc1f1dcf176bb748ac2bbe14 WatchSource:0}: Error finding container 506dd8210ec1842c76ef3ff8e406daeb9b4e4356fc1f1dcf176bb748ac2bbe14: Status 404 returned error can't find the container with id 506dd8210ec1842c76ef3ff8e406daeb9b4e4356fc1f1dcf176bb748ac2bbe14
Sep 16 12:54:58.759650 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:54:58.759605    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" event={"ID":"52e5b035-4654-46a6-a37c-957aa9c5dfb5","Type":"ContainerStarted","Data":"506dd8210ec1842c76ef3ff8e406daeb9b4e4356fc1f1dcf176bb748ac2bbe14"}
Sep 16 12:55:00.714309 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:00.714277    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 12:55:00.772680 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:00.772649    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" event={"ID":"52e5b035-4654-46a6-a37c-957aa9c5dfb5","Type":"ContainerStarted","Data":"2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a"}
Sep 16 12:55:00.772875 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:00.772758    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:55:00.790804 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:00.790756    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" podStartSLOduration=2.5612764180000003 podStartE2EDuration="4.790741101s" totalImagesPullingTime="2.229464683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:00.789806694 +0000 UTC m=+1002.873249731" watchObservedRunningTime="2026-09-16 12:55:00.790741101 +0000 UTC m=+1002.874184115"
Sep 16 12:55:01.719486 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:01.719454    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 12:55:02.166878 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:02.166845    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:55:06.783571 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:06.783541    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:55:07.753388 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:07.753358    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:55:07.753558 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:07.753423    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:55:10.088904 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.088870    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"]
Sep 16 12:55:10.092615 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.092592    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.094656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.094624    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-89d6bf-dockercfg-xvwsq\""
Sep 16 12:55:10.094882 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.094862    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89d6bf\""
Sep 16 12:55:10.101883 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.101863    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"]
Sep 16 12:55:10.219364 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.219314    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqk88\" (UniqueName: \"kubernetes.io/projected/a958870d-94d9-428a-a03a-00f95f3fb111-kube-api-access-kqk88\") pod \"payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.219508 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.219382    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a958870d-94d9-428a-a03a-00f95f3fb111-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.320171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.320135    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqk88\" (UniqueName: \"kubernetes.io/projected/a958870d-94d9-428a-a03a-00f95f3fb111-kube-api-access-kqk88\") pod \"payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.320381 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.320184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a958870d-94d9-428a-a03a-00f95f3fb111-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.320905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.320886    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a958870d-94d9-428a-a03a-00f95f3fb111-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.330268 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.330244    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqk88\" (UniqueName: \"kubernetes.io/projected/a958870d-94d9-428a-a03a-00f95f3fb111-kube-api-access-kqk88\") pod \"payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.403057 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.402970    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.676916 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.676895    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"]
Sep 16 12:55:10.679297 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:55:10.679268    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda958870d_94d9_428a_a03a_00f95f3fb111.slice/crio-d2a0727a4943b4066c6fcfc49bdabcc860db343d463e6cf6527f9987095014ef WatchSource:0}: Error finding container d2a0727a4943b4066c6fcfc49bdabcc860db343d463e6cf6527f9987095014ef: Status 404 returned error can't find the container with id d2a0727a4943b4066c6fcfc49bdabcc860db343d463e6cf6527f9987095014ef
Sep 16 12:55:10.826215 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.826181    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2" event={"ID":"a958870d-94d9-428a-a03a-00f95f3fb111","Type":"ContainerStarted","Data":"41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843"}
Sep 16 12:55:10.826408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.826239    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2" event={"ID":"a958870d-94d9-428a-a03a-00f95f3fb111","Type":"ContainerStarted","Data":"d2a0727a4943b4066c6fcfc49bdabcc860db343d463e6cf6527f9987095014ef"}
Sep 16 12:55:10.826408 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.826276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:10.849343 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:10.844649    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2" podStartSLOduration=0.844630922 podStartE2EDuration="844.630922ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:10.84174017 +0000 UTC m=+1012.925183201" watchObservedRunningTime="2026-09-16 12:55:10.844630922 +0000 UTC m=+1012.928073937"
Sep 16 12:55:21.833198 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:21.833167    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 12:55:21.918625 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:21.918594    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"]
Sep 16 12:55:21.919064 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:21.919027    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" podUID="52e5b035-4654-46a6-a37c-957aa9c5dfb5" containerName="maas-api" containerID="cri-o://2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a" gracePeriod=30
Sep 16 12:55:22.163915 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.163888    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:55:22.332912 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.331785    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8slxc\" (UniqueName: \"kubernetes.io/projected/52e5b035-4654-46a6-a37c-957aa9c5dfb5-kube-api-access-8slxc\") pod \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") "
Sep 16 12:55:22.332912 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.331868    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls\") pod \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") "
Sep 16 12:55:22.332912 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.331912    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5b035-4654-46a6-a37c-957aa9c5dfb5-openshift-service-ca\") pod \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") "
Sep 16 12:55:22.332912 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.332012    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-metrics-certs\") pod \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\" (UID: \"52e5b035-4654-46a6-a37c-957aa9c5dfb5\") "
Sep 16 12:55:22.334171 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.334101    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e5b035-4654-46a6-a37c-957aa9c5dfb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "52e5b035-4654-46a6-a37c-957aa9c5dfb5" (UID: "52e5b035-4654-46a6-a37c-957aa9c5dfb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:55:22.340276 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.340248    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "52e5b035-4654-46a6-a37c-957aa9c5dfb5" (UID: "52e5b035-4654-46a6-a37c-957aa9c5dfb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:22.340428 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.340241    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "52e5b035-4654-46a6-a37c-957aa9c5dfb5" (UID: "52e5b035-4654-46a6-a37c-957aa9c5dfb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:55:22.340428 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.340297    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e5b035-4654-46a6-a37c-957aa9c5dfb5-kube-api-access-8slxc" (OuterVolumeSpecName: "kube-api-access-8slxc") pod "52e5b035-4654-46a6-a37c-957aa9c5dfb5" (UID: "52e5b035-4654-46a6-a37c-957aa9c5dfb5"). InnerVolumeSpecName "kube-api-access-8slxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:55:22.433594 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.433567    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8slxc\" (UniqueName: \"kubernetes.io/projected/52e5b035-4654-46a6-a37c-957aa9c5dfb5-kube-api-access-8slxc\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:55:22.433594 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.433592    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:55:22.433747 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.433601    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52e5b035-4654-46a6-a37c-957aa9c5dfb5-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:55:22.433747 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.433610    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52e5b035-4654-46a6-a37c-957aa9c5dfb5-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:55:22.885630 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.885541    2559 generic.go:356] "Generic (PLEG): container finished" podID="52e5b035-4654-46a6-a37c-957aa9c5dfb5" containerID="2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a" exitCode=0
Sep 16 12:55:22.885630 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.885596    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"
Sep 16 12:55:22.886060 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.885624    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" event={"ID":"52e5b035-4654-46a6-a37c-957aa9c5dfb5","Type":"ContainerDied","Data":"2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a"}
Sep 16 12:55:22.886060 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.885676    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8" event={"ID":"52e5b035-4654-46a6-a37c-957aa9c5dfb5","Type":"ContainerDied","Data":"506dd8210ec1842c76ef3ff8e406daeb9b4e4356fc1f1dcf176bb748ac2bbe14"}
Sep 16 12:55:22.886060 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.885698    2559 scope.go:117] "RemoveContainer" containerID="2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a"
Sep 16 12:55:22.895456 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.895437    2559 scope.go:117] "RemoveContainer" containerID="2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a"
Sep 16 12:55:22.895713 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:55:22.895695    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a\": container with ID starting with 2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a not found: ID does not exist" containerID="2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a"
Sep 16 12:55:22.895765 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.895721    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a"} err="failed to get container status \"2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a\": rpc error: code = NotFound desc = could not find container \"2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a\": container with ID starting with 2592bf3bc809071d06faee1763f2fb52138f8d87187104912e5040a560033f7a not found: ID does not exist"
Sep 16 12:55:22.923193 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.923169    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"]
Sep 16 12:55:22.936032 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:22.936007    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-79dbf9e9-968f965f-rdkw8"]
Sep 16 12:55:24.412515 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:24.412482    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52e5b035-4654-46a6-a37c-957aa9c5dfb5" path="/var/lib/kubelet/pods/52e5b035-4654-46a6-a37c-957aa9c5dfb5/volumes"
Sep 16 12:55:24.749102 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:24.748991    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:55:30.111393 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.111355    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"]
Sep 16 12:55:30.112025 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.111946    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52e5b035-4654-46a6-a37c-957aa9c5dfb5" containerName="maas-api"
Sep 16 12:55:30.112025 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.111966    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="52e5b035-4654-46a6-a37c-957aa9c5dfb5" containerName="maas-api"
Sep 16 12:55:30.112197 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.112071    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="52e5b035-4654-46a6-a37c-957aa9c5dfb5" containerName="maas-api"
Sep 16 12:55:30.119747 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.119729    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.123097 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.123074    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e3535999-serving-cert\""
Sep 16 12:55:30.123293 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.123273    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:55:30.123416 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.123334    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:55:30.123416 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.123360    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:55:30.123416 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.123392    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nm49v\""
Sep 16 12:55:30.128682 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.128661    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"]
Sep 16 12:55:30.220700 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.220671    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-maas-api-tls\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.220879 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.220768    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8k4r\" (UniqueName: \"kubernetes.io/projected/655ca2d7-0425-4b4f-aa78-093afd9addd9-kube-api-access-n8k4r\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.220879 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.220860    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655ca2d7-0425-4b4f-aa78-093afd9addd9-openshift-service-ca\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.220966 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.220889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-metrics-certs\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.321761 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.321720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8k4r\" (UniqueName: \"kubernetes.io/projected/655ca2d7-0425-4b4f-aa78-093afd9addd9-kube-api-access-n8k4r\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.321948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.321801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655ca2d7-0425-4b4f-aa78-093afd9addd9-openshift-service-ca\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.321948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.321824    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-metrics-certs\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.321948 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.321865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-maas-api-tls\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.322696 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.322579    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655ca2d7-0425-4b4f-aa78-093afd9addd9-openshift-service-ca\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.324547 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.324518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-maas-api-tls\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.324662 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.324594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-metrics-certs\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.334886 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.334860    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8k4r\" (UniqueName: \"kubernetes.io/projected/655ca2d7-0425-4b4f-aa78-093afd9addd9-kube-api-access-n8k4r\") pod \"maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.431585 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.431555    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:30.714055 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.714030    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"]
Sep 16 12:55:30.715724 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:55:30.715699    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655ca2d7_0425_4b4f_aa78_093afd9addd9.slice/crio-7c4ca002b5d19d03a0d177091adca0dd05f69b91351338f6cc71a64cac090943 WatchSource:0}: Error finding container 7c4ca002b5d19d03a0d177091adca0dd05f69b91351338f6cc71a64cac090943: Status 404 returned error can't find the container with id 7c4ca002b5d19d03a0d177091adca0dd05f69b91351338f6cc71a64cac090943
Sep 16 12:55:30.933441 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.933396    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww" event={"ID":"655ca2d7-0425-4b4f-aa78-093afd9addd9","Type":"ContainerStarted","Data":"7c4ca002b5d19d03a0d177091adca0dd05f69b91351338f6cc71a64cac090943"}
Sep 16 12:55:30.976128 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.976048    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"]
Sep 16 12:55:30.981585 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.981565    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:30.983799 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.983780    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e3535999\""
Sep 16 12:55:30.983915 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.983782    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e3535999-dockercfg-htfp4\""
Sep 16 12:55:30.991146 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:30.991120    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"]
Sep 16 12:55:31.026543 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.026521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24hf\" (UniqueName: \"kubernetes.io/projected/a6bf3af7-da05-4caf-958d-5114892abea8-kube-api-access-x24hf\") pod \"payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.026657 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.026569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bf3af7-da05-4caf-958d-5114892abea8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.127110 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.127078    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24hf\" (UniqueName: \"kubernetes.io/projected/a6bf3af7-da05-4caf-958d-5114892abea8-kube-api-access-x24hf\") pod \"payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.127467 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.127117    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bf3af7-da05-4caf-958d-5114892abea8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.127840 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.127821    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bf3af7-da05-4caf-958d-5114892abea8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.135928 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.135910    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24hf\" (UniqueName: \"kubernetes.io/projected/a6bf3af7-da05-4caf-958d-5114892abea8-kube-api-access-x24hf\") pod \"payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") " pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.293550 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.293478    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.567657 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.567632    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"]
Sep 16 12:55:31.569934 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:55:31.569905    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bf3af7_da05_4caf_958d_5114892abea8.slice/crio-95d23196d946b0880ff43e1d6dba27ec52e1cb9ec6cbca1d1288e255d7957d85 WatchSource:0}: Error finding container 95d23196d946b0880ff43e1d6dba27ec52e1cb9ec6cbca1d1288e255d7957d85: Status 404 returned error can't find the container with id 95d23196d946b0880ff43e1d6dba27ec52e1cb9ec6cbca1d1288e255d7957d85
Sep 16 12:55:31.941041 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.940999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww" event={"ID":"655ca2d7-0425-4b4f-aa78-093afd9addd9","Type":"ContainerStarted","Data":"a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33"}
Sep 16 12:55:31.941253 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.941085    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:31.942705 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.942682    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf" event={"ID":"a6bf3af7-da05-4caf-958d-5114892abea8","Type":"ContainerStarted","Data":"0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e"}
Sep 16 12:55:31.942812 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.942721    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf" event={"ID":"a6bf3af7-da05-4caf-958d-5114892abea8","Type":"ContainerStarted","Data":"95d23196d946b0880ff43e1d6dba27ec52e1cb9ec6cbca1d1288e255d7957d85"}
Sep 16 12:55:31.942812 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.942766    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:31.962889 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.962835    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww" podStartSLOduration=1.669625176 podStartE2EDuration="1.96281495s" totalImagesPullingTime="293.189774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:31.962517609 +0000 UTC m=+1034.045960645" watchObservedRunningTime="2026-09-16 12:55:31.96281495 +0000 UTC m=+1034.046257966"
Sep 16 12:55:31.982073 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:31.982032    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf" podStartSLOduration=1.982017339 podStartE2EDuration="1.982017339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:55:31.980741814 +0000 UTC m=+1034.064184849" watchObservedRunningTime="2026-09-16 12:55:31.982017339 +0000 UTC m=+1034.065460376"
Sep 16 12:55:37.953699 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:37.953672    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:55:38.055754 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:38.055705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:55:42.950693 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:42.950662    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:55:58.854485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:55:58.854451    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:56:08.159502 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.159461    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"]
Sep 16 12:56:08.159923 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.159720    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww" podUID="655ca2d7-0425-4b4f-aa78-093afd9addd9" containerName="maas-api" containerID="cri-o://a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33" gracePeriod=30
Sep 16 12:56:08.223531 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.223482    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"]
Sep 16 12:56:08.224335 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.224284    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf" podUID="a6bf3af7-da05-4caf-958d-5114892abea8" containerName="payload-processing" containerID="cri-o://0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e" gracePeriod=30
Sep 16 12:56:08.492644 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.492620    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:56:08.520385 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.520359    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:56:08.671986 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.671962    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-metrics-certs\") pod \"655ca2d7-0425-4b4f-aa78-093afd9addd9\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") "
Sep 16 12:56:08.672172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.671999    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655ca2d7-0425-4b4f-aa78-093afd9addd9-openshift-service-ca\") pod \"655ca2d7-0425-4b4f-aa78-093afd9addd9\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") "
Sep 16 12:56:08.672172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.672035    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-maas-api-tls\") pod \"655ca2d7-0425-4b4f-aa78-093afd9addd9\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") "
Sep 16 12:56:08.672172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.672063    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x24hf\" (UniqueName: \"kubernetes.io/projected/a6bf3af7-da05-4caf-958d-5114892abea8-kube-api-access-x24hf\") pod \"a6bf3af7-da05-4caf-958d-5114892abea8\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") "
Sep 16 12:56:08.672172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.672108    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bf3af7-da05-4caf-958d-5114892abea8-plugins-config-volume\") pod \"a6bf3af7-da05-4caf-958d-5114892abea8\" (UID: \"a6bf3af7-da05-4caf-958d-5114892abea8\") "
Sep 16 12:56:08.672172 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.672139    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8k4r\" (UniqueName: \"kubernetes.io/projected/655ca2d7-0425-4b4f-aa78-093afd9addd9-kube-api-access-n8k4r\") pod \"655ca2d7-0425-4b4f-aa78-093afd9addd9\" (UID: \"655ca2d7-0425-4b4f-aa78-093afd9addd9\") "
Sep 16 12:56:08.672506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.672477    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655ca2d7-0425-4b4f-aa78-093afd9addd9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "655ca2d7-0425-4b4f-aa78-093afd9addd9" (UID: "655ca2d7-0425-4b4f-aa78-093afd9addd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:08.672582 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.672555    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bf3af7-da05-4caf-958d-5114892abea8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6bf3af7-da05-4caf-958d-5114892abea8" (UID: "a6bf3af7-da05-4caf-958d-5114892abea8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:08.674454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.674429    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "655ca2d7-0425-4b4f-aa78-093afd9addd9" (UID: "655ca2d7-0425-4b4f-aa78-093afd9addd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:56:08.674576 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.674519    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bf3af7-da05-4caf-958d-5114892abea8-kube-api-access-x24hf" (OuterVolumeSpecName: "kube-api-access-x24hf") pod "a6bf3af7-da05-4caf-958d-5114892abea8" (UID: "a6bf3af7-da05-4caf-958d-5114892abea8"). InnerVolumeSpecName "kube-api-access-x24hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:08.674867 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.674849    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "655ca2d7-0425-4b4f-aa78-093afd9addd9" (UID: "655ca2d7-0425-4b4f-aa78-093afd9addd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:56:08.674867 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.674852    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655ca2d7-0425-4b4f-aa78-093afd9addd9-kube-api-access-n8k4r" (OuterVolumeSpecName: "kube-api-access-n8k4r") pod "655ca2d7-0425-4b4f-aa78-093afd9addd9" (UID: "655ca2d7-0425-4b4f-aa78-093afd9addd9"). InnerVolumeSpecName "kube-api-access-n8k4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:08.773802 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.773714    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bf3af7-da05-4caf-958d-5114892abea8-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:08.773802 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.773746    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8k4r\" (UniqueName: \"kubernetes.io/projected/655ca2d7-0425-4b4f-aa78-093afd9addd9-kube-api-access-n8k4r\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:08.773802 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.773757    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:08.773802 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.773767    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/655ca2d7-0425-4b4f-aa78-093afd9addd9-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:08.773802 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.773775    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/655ca2d7-0425-4b4f-aa78-093afd9addd9-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:08.773802 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:08.773784    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24hf\" (UniqueName: \"kubernetes.io/projected/a6bf3af7-da05-4caf-958d-5114892abea8-kube-api-access-x24hf\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:09.130232 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.130140    2559 generic.go:356] "Generic (PLEG): container finished" podID="655ca2d7-0425-4b4f-aa78-093afd9addd9" containerID="a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33" exitCode=0
Sep 16 12:56:09.130232 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.130194    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"
Sep 16 12:56:09.130540 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.130233    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww" event={"ID":"655ca2d7-0425-4b4f-aa78-093afd9addd9","Type":"ContainerDied","Data":"a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33"}
Sep 16 12:56:09.130540 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.130294    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww" event={"ID":"655ca2d7-0425-4b4f-aa78-093afd9addd9","Type":"ContainerDied","Data":"7c4ca002b5d19d03a0d177091adca0dd05f69b91351338f6cc71a64cac090943"}
Sep 16 12:56:09.130540 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.130332    2559 scope.go:117] "RemoveContainer" containerID="a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33"
Sep 16 12:56:09.131658 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.131638    2559 generic.go:356] "Generic (PLEG): container finished" podID="a6bf3af7-da05-4caf-958d-5114892abea8" containerID="0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e" exitCode=0
Sep 16 12:56:09.131763 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.131689    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"
Sep 16 12:56:09.131763 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.131731    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf" event={"ID":"a6bf3af7-da05-4caf-958d-5114892abea8","Type":"ContainerDied","Data":"0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e"}
Sep 16 12:56:09.131852 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.131771    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf" event={"ID":"a6bf3af7-da05-4caf-958d-5114892abea8","Type":"ContainerDied","Data":"95d23196d946b0880ff43e1d6dba27ec52e1cb9ec6cbca1d1288e255d7957d85"}
Sep 16 12:56:09.141075 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.140990    2559 scope.go:117] "RemoveContainer" containerID="a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33"
Sep 16 12:56:09.141421 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:56:09.141395    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33\": container with ID starting with a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33 not found: ID does not exist" containerID="a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33"
Sep 16 12:56:09.141505 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.141432    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33"} err="failed to get container status \"a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33\": rpc error: code = NotFound desc = could not find container \"a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33\": container with ID starting with a1ca40c3a0610f1b7214921c6ae0ae5065701cc206a4888d8f2b21008c732b33 not found: ID does not exist"
Sep 16 12:56:09.141505 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.141451    2559 scope.go:117] "RemoveContainer" containerID="0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e"
Sep 16 12:56:09.149506 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.149487    2559 scope.go:117] "RemoveContainer" containerID="0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e"
Sep 16 12:56:09.149769 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:56:09.149749    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e\": container with ID starting with 0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e not found: ID does not exist" containerID="0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e"
Sep 16 12:56:09.149833 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.149775    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e"} err="failed to get container status \"0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e\": rpc error: code = NotFound desc = could not find container \"0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e\": container with ID starting with 0e9842ca612f74ed402e24637d6cf9b1350bd3e088bf2006fbb3823b9192cd6e not found: ID does not exist"
Sep 16 12:56:09.153623 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.153601    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"]
Sep 16 12:56:09.159661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.159641    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e3535999-8d5896d7c-2fkww"]
Sep 16 12:56:09.177382 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.177304    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"]
Sep 16 12:56:09.180856 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:09.180836    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-e3535999-5c55c5ff9d-jxrqf"]
Sep 16 12:56:10.411903 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:10.411874    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="655ca2d7-0425-4b4f-aa78-093afd9addd9" path="/var/lib/kubelet/pods/655ca2d7-0425-4b4f-aa78-093afd9addd9/volumes"
Sep 16 12:56:10.412268 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:10.412232    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6bf3af7-da05-4caf-958d-5114892abea8" path="/var/lib/kubelet/pods/a6bf3af7-da05-4caf-958d-5114892abea8/volumes"
Sep 16 12:56:13.954297 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:13.954265    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:56:14.251289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251210    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"]
Sep 16 12:56:14.251609 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251595    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="655ca2d7-0425-4b4f-aa78-093afd9addd9" containerName="maas-api"
Sep 16 12:56:14.251676 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251609    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="655ca2d7-0425-4b4f-aa78-093afd9addd9" containerName="maas-api"
Sep 16 12:56:14.251676 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251628    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6bf3af7-da05-4caf-958d-5114892abea8" containerName="payload-processing"
Sep 16 12:56:14.251676 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251634    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6bf3af7-da05-4caf-958d-5114892abea8" containerName="payload-processing"
Sep 16 12:56:14.251830 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251694    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="655ca2d7-0425-4b4f-aa78-093afd9addd9" containerName="maas-api"
Sep 16 12:56:14.251830 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.251702    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6bf3af7-da05-4caf-958d-5114892abea8" containerName="payload-processing"
Sep 16 12:56:14.256219 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.256197    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.258724 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.258698    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-19dd11cf-serving-cert\""
Sep 16 12:56:14.259440 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.259416    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:56:14.259440 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.259427    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nm49v\""
Sep 16 12:56:14.259599 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.259451    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:56:14.259599 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.259425    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:56:14.267952 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.267928    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"]
Sep 16 12:56:14.326442 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.326411    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-openshift-service-ca\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.326586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.326478    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.326586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.326521    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-metrics-certs\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.326586 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.326558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5nlq\" (UniqueName: \"kubernetes.io/projected/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-kube-api-access-k5nlq\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.427929 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.427899    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-openshift-service-ca\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.428116 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.427951    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.428116 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.427967    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-metrics-certs\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.428116 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:56:14.428050    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-serving-cert: secret "maas-api-e2e-ait-19dd11cf-serving-cert" not found
Sep 16 12:56:14.428116 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.428072    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5nlq\" (UniqueName: \"kubernetes.io/projected/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-kube-api-access-k5nlq\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.428116 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:56:14.428109    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls podName:a01302d1-a05f-4739-b6fa-e9ef3cbbfccc nodeName:}" failed. No retries permitted until 2026-09-16 12:56:14.928093128 +0000 UTC m=+1077.011536120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls") pod "maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" (UID: "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc") : secret "maas-api-e2e-ait-19dd11cf-serving-cert" not found
Sep 16 12:56:14.428667 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.428645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-openshift-service-ca\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.430483 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.430467    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-metrics-certs\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.439285 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.439266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5nlq\" (UniqueName: \"kubernetes.io/projected/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-kube-api-access-k5nlq\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.897142 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.897111    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"]
Sep 16 12:56:14.900584 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.900569    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:14.902980 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.902950    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-19dd11cf\""
Sep 16 12:56:14.903128 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.903104    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-19dd11cf-dockercfg-5k7rx\""
Sep 16 12:56:14.913960 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.913933    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"]
Sep 16 12:56:14.932103 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.932074    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:14.932244 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.932111    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-plugins-config-volume\") pod \"payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") " pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:14.932244 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.932139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsxjc\" (UniqueName: \"kubernetes.io/projected/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-kube-api-access-tsxjc\") pod \"payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") " pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:14.934641 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:14.934614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls\") pod \"maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:15.033377 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.033340    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-plugins-config-volume\") pod \"payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") " pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:15.033737 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.033393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsxjc\" (UniqueName: \"kubernetes.io/projected/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-kube-api-access-tsxjc\") pod \"payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") " pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:15.034114 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.034087    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-plugins-config-volume\") pod \"payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") " pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:15.046509 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.046481    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsxjc\" (UniqueName: \"kubernetes.io/projected/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-kube-api-access-tsxjc\") pod \"payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") " pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:15.168424 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.168387    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:15.212415 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.212380    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:15.441595 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.441574    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"]
Sep 16 12:56:15.443489 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:56:15.443462    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01302d1_a05f_4739_b6fa_e9ef3cbbfccc.slice/crio-9f0a10c6a2739a29a804155c2c53d4b8996204d44f3e754cd72e85fc91f3a111 WatchSource:0}: Error finding container 9f0a10c6a2739a29a804155c2c53d4b8996204d44f3e754cd72e85fc91f3a111: Status 404 returned error can't find the container with id 9f0a10c6a2739a29a804155c2c53d4b8996204d44f3e754cd72e85fc91f3a111
Sep 16 12:56:15.495908 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:15.495884    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"]
Sep 16 12:56:15.499059 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:56:15.499037    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eec312_8d7b_48a6_b18b_f7d4e394bd18.slice/crio-ef80176a8edc9ee05e259bc778c1e5c52d4f0a1c5d5410e019a228524700e7a0 WatchSource:0}: Error finding container ef80176a8edc9ee05e259bc778c1e5c52d4f0a1c5d5410e019a228524700e7a0: Status 404 returned error can't find the container with id ef80176a8edc9ee05e259bc778c1e5c52d4f0a1c5d5410e019a228524700e7a0
Sep 16 12:56:16.169026 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.168932    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff" event={"ID":"c7eec312-8d7b-48a6-b18b-f7d4e394bd18","Type":"ContainerStarted","Data":"cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09"}
Sep 16 12:56:16.169426 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.169113    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:16.169426 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.169145    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff" event={"ID":"c7eec312-8d7b-48a6-b18b-f7d4e394bd18","Type":"ContainerStarted","Data":"ef80176a8edc9ee05e259bc778c1e5c52d4f0a1c5d5410e019a228524700e7a0"}
Sep 16 12:56:16.170416 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.170393    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" event={"ID":"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc","Type":"ContainerStarted","Data":"fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d"}
Sep 16 12:56:16.170522 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.170435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" event={"ID":"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc","Type":"ContainerStarted","Data":"9f0a10c6a2739a29a804155c2c53d4b8996204d44f3e754cd72e85fc91f3a111"}
Sep 16 12:56:16.170522 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.170471    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:16.188481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.188438    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff" podStartSLOduration=2.188423302 podStartE2EDuration="2.188423302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:16.186516285 +0000 UTC m=+1078.269959330" watchObservedRunningTime="2026-09-16 12:56:16.188423302 +0000 UTC m=+1078.271866316"
Sep 16 12:56:16.204226 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:16.204184    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" podStartSLOduration=1.7297341880000001 podStartE2EDuration="2.204172834s" totalImagesPullingTime="474.438646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:16.203453174 +0000 UTC m=+1078.286896220" watchObservedRunningTime="2026-09-16 12:56:16.204172834 +0000 UTC m=+1078.287615847"
Sep 16 12:56:22.181791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:22.181765    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:27.178384 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:27.178354    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:31.761563 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:31.761531    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:56:43.824461 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:43.824424    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"]
Sep 16 12:56:43.824970 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:43.824665    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" podUID="a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" containerName="maas-api" containerID="cri-o://fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d" gracePeriod=30
Sep 16 12:56:43.887214 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:43.887167    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"]
Sep 16 12:56:43.887504 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:43.887461    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff" podUID="c7eec312-8d7b-48a6-b18b-f7d4e394bd18" containerName="payload-processing" containerID="cri-o://cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09" gracePeriod=30
Sep 16 12:56:44.136393 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.136287    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:44.154791 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.154731    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:44.308149 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.308115    2559 generic.go:356] "Generic (PLEG): container finished" podID="c7eec312-8d7b-48a6-b18b-f7d4e394bd18" containerID="cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09" exitCode=0
Sep 16 12:56:44.308377 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.308167    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"
Sep 16 12:56:44.308377 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.308222    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff" event={"ID":"c7eec312-8d7b-48a6-b18b-f7d4e394bd18","Type":"ContainerDied","Data":"cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09"}
Sep 16 12:56:44.308377 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.308276    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff" event={"ID":"c7eec312-8d7b-48a6-b18b-f7d4e394bd18","Type":"ContainerDied","Data":"ef80176a8edc9ee05e259bc778c1e5c52d4f0a1c5d5410e019a228524700e7a0"}
Sep 16 12:56:44.308377 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.308308    2559 scope.go:117] "RemoveContainer" containerID="cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09"
Sep 16 12:56:44.309557 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.309538    2559 generic.go:356] "Generic (PLEG): container finished" podID="a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" containerID="fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d" exitCode=0
Sep 16 12:56:44.309656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.309580    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" event={"ID":"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc","Type":"ContainerDied","Data":"fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d"}
Sep 16 12:56:44.309656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.309620    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc" event={"ID":"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc","Type":"ContainerDied","Data":"9f0a10c6a2739a29a804155c2c53d4b8996204d44f3e754cd72e85fc91f3a111"}
Sep 16 12:56:44.309656 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.309586    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"
Sep 16 12:56:44.317120 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.317100    2559 scope.go:117] "RemoveContainer" containerID="cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09"
Sep 16 12:56:44.317403 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:56:44.317384    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09\": container with ID starting with cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09 not found: ID does not exist" containerID="cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09"
Sep 16 12:56:44.317485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.317416    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09"} err="failed to get container status \"cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09\": rpc error: code = NotFound desc = could not find container \"cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09\": container with ID starting with cd45af10f518c278c8be9b8002d9565b479463ffd6bfb73783f8f25b80a18b09 not found: ID does not exist"
Sep 16 12:56:44.317485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.317439    2559 scope.go:117] "RemoveContainer" containerID="fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d"
Sep 16 12:56:44.320607 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.320587    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsxjc\" (UniqueName: \"kubernetes.io/projected/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-kube-api-access-tsxjc\") pod \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") "
Sep 16 12:56:44.320695 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.320635    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-plugins-config-volume\") pod \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\" (UID: \"c7eec312-8d7b-48a6-b18b-f7d4e394bd18\") "
Sep 16 12:56:44.320759 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.320707    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5nlq\" (UniqueName: \"kubernetes.io/projected/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-kube-api-access-k5nlq\") pod \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") "
Sep 16 12:56:44.320759 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.320750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls\") pod \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") "
Sep 16 12:56:44.320872 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.320806    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-metrics-certs\") pod \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") "
Sep 16 12:56:44.320872 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.320862    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-openshift-service-ca\") pod \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\" (UID: \"a01302d1-a05f-4739-b6fa-e9ef3cbbfccc\") "
Sep 16 12:56:44.321190 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.321087    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7eec312-8d7b-48a6-b18b-f7d4e394bd18" (UID: "c7eec312-8d7b-48a6-b18b-f7d4e394bd18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:44.321276 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.321255    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" (UID: "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:56:44.322882 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.322861    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-kube-api-access-tsxjc" (OuterVolumeSpecName: "kube-api-access-tsxjc") pod "c7eec312-8d7b-48a6-b18b-f7d4e394bd18" (UID: "c7eec312-8d7b-48a6-b18b-f7d4e394bd18"). InnerVolumeSpecName "kube-api-access-tsxjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:44.323024 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.322997    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-kube-api-access-k5nlq" (OuterVolumeSpecName: "kube-api-access-k5nlq") pod "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" (UID: "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc"). InnerVolumeSpecName "kube-api-access-k5nlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:56:44.323139 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.323065    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" (UID: "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:56:44.323236 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.323220    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" (UID: "a01302d1-a05f-4739-b6fa-e9ef3cbbfccc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:56:44.326250 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.326232    2559 scope.go:117] "RemoveContainer" containerID="fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d"
Sep 16 12:56:44.326547 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:56:44.326531    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d\": container with ID starting with fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d not found: ID does not exist" containerID="fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d"
Sep 16 12:56:44.326600 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.326557    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d"} err="failed to get container status \"fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d\": rpc error: code = NotFound desc = could not find container \"fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d\": container with ID starting with fcd8f9ceadf668468382bf4232b23265ebc83ae8e89db6463ec0350d81dc792d not found: ID does not exist"
Sep 16 12:56:44.421827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.421689    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.421827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.421715    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.421827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.421727    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsxjc\" (UniqueName: \"kubernetes.io/projected/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-kube-api-access-tsxjc\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.421827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.421736    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7eec312-8d7b-48a6-b18b-f7d4e394bd18-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.421827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.421745    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5nlq\" (UniqueName: \"kubernetes.io/projected/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-kube-api-access-k5nlq\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.421827 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.421754    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:56:44.627169 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.627137    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"]
Sep 16 12:56:44.631065 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.631043    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19dd11cf-849f4bd677-brwff"]
Sep 16 12:56:44.639943 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.639921    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"]
Sep 16 12:56:44.645253 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:44.645230    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19dd11cf-5b455684f9-jsgnc"]
Sep 16 12:56:46.411546 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:46.411514    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" path="/var/lib/kubelet/pods/a01302d1-a05f-4739-b6fa-e9ef3cbbfccc/volumes"
Sep 16 12:56:46.411924 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:46.411887    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7eec312-8d7b-48a6-b18b-f7d4e394bd18" path="/var/lib/kubelet/pods/c7eec312-8d7b-48a6-b18b-f7d4e394bd18/volumes"
Sep 16 12:56:50.935969 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.935935    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"]
Sep 16 12:56:50.936371 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.936336    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7eec312-8d7b-48a6-b18b-f7d4e394bd18" containerName="payload-processing"
Sep 16 12:56:50.936371 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.936348    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7eec312-8d7b-48a6-b18b-f7d4e394bd18" containerName="payload-processing"
Sep 16 12:56:50.936454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.936372    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" containerName="maas-api"
Sep 16 12:56:50.936454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.936379    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" containerName="maas-api"
Sep 16 12:56:50.936454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.936439    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a01302d1-a05f-4739-b6fa-e9ef3cbbfccc" containerName="maas-api"
Sep 16 12:56:50.936454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.936449    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7eec312-8d7b-48a6-b18b-f7d4e394bd18" containerName="payload-processing"
Sep 16 12:56:50.939692 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.939676    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:50.942044 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.942025    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"openshift-service-ca.crt\""
Sep 16 12:56:50.942895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.942869    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"default-dockercfg-8mkcq\""
Sep 16 12:56:50.942895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.942881    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"kube-root-ca.crt\""
Sep 16 12:56:50.942895 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.942876    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"simulator-w4-7d0256-kserve-self-signed-certs\""
Sep 16 12:56:50.949500 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.949476    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"]
Sep 16 12:56:50.986454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.986424    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-model-cache\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:50.986608 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.986474    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095ee0f2-9efb-44ce-a883-751f66521957-tls-certs\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:50.986608 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.986541    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-dshm\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:50.986608 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.986576    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-home\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:50.986713 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.986609    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7td\" (UniqueName: \"kubernetes.io/projected/095ee0f2-9efb-44ce-a883-751f66521957-kube-api-access-6n7td\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:50.986713 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:50.986649    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-tmp-dir\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087099 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087061    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7td\" (UniqueName: \"kubernetes.io/projected/095ee0f2-9efb-44ce-a883-751f66521957-kube-api-access-6n7td\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-tmp-dir\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-model-cache\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095ee0f2-9efb-44ce-a883-751f66521957-tls-certs\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087224    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-dshm\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087289 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087249    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-home\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087633 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087607    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-tmp-dir\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087692 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087628    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-model-cache\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.087692 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.087652    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-home\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.089726 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.089706    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-dshm\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.089995 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.089977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095ee0f2-9efb-44ce-a883-751f66521957-tls-certs\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.095452 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.095430    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7td\" (UniqueName: \"kubernetes.io/projected/095ee0f2-9efb-44ce-a883-751f66521957-kube-api-access-6n7td\") pod \"simulator-w4-7d0256-kserve-74467bbf59-xx7t6\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") " pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.249107 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.249007    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:56:51.521904 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.521878    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"]
Sep 16 12:56:51.523726 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:56:51.523684    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095ee0f2_9efb_44ce_a883_751f66521957.slice/crio-8a7487d9ba5f4da642a6371a7f731e3776f62e929d6a82ce29598caebc074fbd WatchSource:0}: Error finding container 8a7487d9ba5f4da642a6371a7f731e3776f62e929d6a82ce29598caebc074fbd: Status 404 returned error can't find the container with id 8a7487d9ba5f4da642a6371a7f731e3776f62e929d6a82ce29598caebc074fbd
Sep 16 12:56:51.748455 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:51.748416    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:56:52.352425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:52.352385    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6" event={"ID":"095ee0f2-9efb-44ce-a883-751f66521957","Type":"ContainerStarted","Data":"020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429"}
Sep 16 12:56:52.352425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:52.352431    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6" event={"ID":"095ee0f2-9efb-44ce-a883-751f66521957","Type":"ContainerStarted","Data":"8a7487d9ba5f4da642a6371a7f731e3776f62e929d6a82ce29598caebc074fbd"}
Sep 16 12:56:52.371938 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:52.371884    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6" podStartSLOduration=2.371869914 podStartE2EDuration="2.371869914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:52.36986662 +0000 UTC m=+1114.453309638" watchObservedRunningTime="2026-09-16 12:56:52.371869914 +0000 UTC m=+1114.455312928"
Sep 16 12:56:53.737108 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.737073    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"]
Sep 16 12:56:53.740766 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.740744    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.743203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.743178    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-d0f5a035-serving-cert\""
Sep 16 12:56:53.743203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.743194    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:56:53.743855 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.743835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:56:53.743989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.743868    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:56:53.743989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.743913    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nm49v\""
Sep 16 12:56:53.750129 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.750105    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"]
Sep 16 12:56:53.813942 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.813907    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36553dbf-5c97-4996-808a-135fa40a79d5-openshift-service-ca\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.813942 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.813943    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-metrics-certs\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.814176 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.813965    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-maas-api-tls\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.814176 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.814017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjx7q\" (UniqueName: \"kubernetes.io/projected/36553dbf-5c97-4996-808a-135fa40a79d5-kube-api-access-sjx7q\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.914742 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.914704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36553dbf-5c97-4996-808a-135fa40a79d5-openshift-service-ca\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.914742 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.914745    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-metrics-certs\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.915056 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.914773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-maas-api-tls\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.915056 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.914834    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjx7q\" (UniqueName: \"kubernetes.io/projected/36553dbf-5c97-4996-808a-135fa40a79d5-kube-api-access-sjx7q\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.915562 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.915532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36553dbf-5c97-4996-808a-135fa40a79d5-openshift-service-ca\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.917553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.917532    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-metrics-certs\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.917553 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.917543    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-maas-api-tls\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:53.924131 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:53.924110    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjx7q\" (UniqueName: \"kubernetes.io/projected/36553dbf-5c97-4996-808a-135fa40a79d5-kube-api-access-sjx7q\") pod \"maas-api-e2e-derive-d0f5a035-57d99c486c-j4585\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:54.053861 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:54.053765    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:54.325626 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:54.325601    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"]
Sep 16 12:56:54.327811 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:56:54.327787    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36553dbf_5c97_4996_808a_135fa40a79d5.slice/crio-50903ec15a863cc0b467f0caaed40a6fa4b779ad0ab2f93986565b37d15dc27e WatchSource:0}: Error finding container 50903ec15a863cc0b467f0caaed40a6fa4b779ad0ab2f93986565b37d15dc27e: Status 404 returned error can't find the container with id 50903ec15a863cc0b467f0caaed40a6fa4b779ad0ab2f93986565b37d15dc27e
Sep 16 12:56:54.362425 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:54.362394    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585" event={"ID":"36553dbf-5c97-4996-808a-135fa40a79d5","Type":"ContainerStarted","Data":"50903ec15a863cc0b467f0caaed40a6fa4b779ad0ab2f93986565b37d15dc27e"}
Sep 16 12:56:55.370378 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:55.370342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585" event={"ID":"36553dbf-5c97-4996-808a-135fa40a79d5","Type":"ContainerStarted","Data":"edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77"}
Sep 16 12:56:55.370780 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:55.370407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:56:55.403439 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:56:55.403388    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585" podStartSLOduration=2.0265328 podStartE2EDuration="2.403371164s" totalImagesPullingTime="376.838364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:56:55.401443407 +0000 UTC m=+1117.484886432" watchObservedRunningTime="2026-09-16 12:56:55.403371164 +0000 UTC m=+1117.486814220"
Sep 16 12:57:01.249216 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:01.249155    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:57:01.249216 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:01.249222    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:57:01.262314 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:01.262284    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:57:01.381485 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:01.381456    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:57:01.413067 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:01.413032    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 12:57:11.352315 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:11.352279    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:57:30.852571 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:30.852540    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"]
Sep 16 12:57:30.852989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:30.852769    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585" podUID="36553dbf-5c97-4996-808a-135fa40a79d5" containerName="maas-api" containerID="cri-o://edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77" gracePeriod=30
Sep 16 12:57:31.110389 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.110288    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:57:31.174905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.174872    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-maas-api-tls\") pod \"36553dbf-5c97-4996-808a-135fa40a79d5\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") "
Sep 16 12:57:31.174905 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.174907    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjx7q\" (UniqueName: \"kubernetes.io/projected/36553dbf-5c97-4996-808a-135fa40a79d5-kube-api-access-sjx7q\") pod \"36553dbf-5c97-4996-808a-135fa40a79d5\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") "
Sep 16 12:57:31.175154 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.174961    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-metrics-certs\") pod \"36553dbf-5c97-4996-808a-135fa40a79d5\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") "
Sep 16 12:57:31.175154 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.175021    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36553dbf-5c97-4996-808a-135fa40a79d5-openshift-service-ca\") pod \"36553dbf-5c97-4996-808a-135fa40a79d5\" (UID: \"36553dbf-5c97-4996-808a-135fa40a79d5\") "
Sep 16 12:57:31.175465 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.175433    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36553dbf-5c97-4996-808a-135fa40a79d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36553dbf-5c97-4996-808a-135fa40a79d5" (UID: "36553dbf-5c97-4996-808a-135fa40a79d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:31.177295 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.177266    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36553dbf-5c97-4996-808a-135fa40a79d5" (UID: "36553dbf-5c97-4996-808a-135fa40a79d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:57:31.177464 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.177311    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36553dbf-5c97-4996-808a-135fa40a79d5" (UID: "36553dbf-5c97-4996-808a-135fa40a79d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:57:31.177464 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.177361    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36553dbf-5c97-4996-808a-135fa40a79d5-kube-api-access-sjx7q" (OuterVolumeSpecName: "kube-api-access-sjx7q") pod "36553dbf-5c97-4996-808a-135fa40a79d5" (UID: "36553dbf-5c97-4996-808a-135fa40a79d5"). InnerVolumeSpecName "kube-api-access-sjx7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:31.276486 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.276456    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:31.276486 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.276479    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36553dbf-5c97-4996-808a-135fa40a79d5-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:31.276486 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.276491    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36553dbf-5c97-4996-808a-135fa40a79d5-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:31.276735 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.276499    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjx7q\" (UniqueName: \"kubernetes.io/projected/36553dbf-5c97-4996-808a-135fa40a79d5-kube-api-access-sjx7q\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:31.552879 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.552846    2559 generic.go:356] "Generic (PLEG): container finished" podID="36553dbf-5c97-4996-808a-135fa40a79d5" containerID="edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77" exitCode=0
Sep 16 12:57:31.553102 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.552906    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"
Sep 16 12:57:31.553102 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.552933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585" event={"ID":"36553dbf-5c97-4996-808a-135fa40a79d5","Type":"ContainerDied","Data":"edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77"}
Sep 16 12:57:31.553102 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.552991    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585" event={"ID":"36553dbf-5c97-4996-808a-135fa40a79d5","Type":"ContainerDied","Data":"50903ec15a863cc0b467f0caaed40a6fa4b779ad0ab2f93986565b37d15dc27e"}
Sep 16 12:57:31.553102 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.553015    2559 scope.go:117] "RemoveContainer" containerID="edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77"
Sep 16 12:57:31.562528 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.562514    2559 scope.go:117] "RemoveContainer" containerID="edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77"
Sep 16 12:57:31.562823 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:57:31.562803    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77\": container with ID starting with edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77 not found: ID does not exist" containerID="edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77"
Sep 16 12:57:31.562921 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.562830    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77"} err="failed to get container status \"edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77\": rpc error: code = NotFound desc = could not find container \"edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77\": container with ID starting with edd8895f7b0e9ab75cdf98888213f8d27862cdb7401cd141ff5a8242f285ff77 not found: ID does not exist"
Sep 16 12:57:31.577065 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.577028    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"]
Sep 16 12:57:31.580902 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:31.580880    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-d0f5a035-57d99c486c-j4585"]
Sep 16 12:57:32.412017 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:32.411985    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36553dbf-5c97-4996-808a-135fa40a79d5" path="/var/lib/kubelet/pods/36553dbf-5c97-4996-808a-135fa40a79d5/volumes"
Sep 16 12:57:33.011206 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:33.011175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:57:56.158409 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:56.158312    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:57:57.965385 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:57.965351    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"]
Sep 16 12:57:57.965811 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:57.965596    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7" podUID="805418a0-de25-4fa3-ae83-434958cc0b5f" containerName="payload-processing" containerID="cri-o://3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5" gracePeriod=30
Sep 16 12:57:57.974311 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:57.973722    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"]
Sep 16 12:57:57.974311 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:57.973956    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v" podUID="7b96a68e-1b91-4a69-adb1-f01b57002d5d" containerName="payload-pre-processing" containerID="cri-o://2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37" gracePeriod=30
Sep 16 12:57:58.245653 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.245625    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:57:58.248205 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.248187    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:57:58.342412 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.342371    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql7rp\" (UniqueName: \"kubernetes.io/projected/7b96a68e-1b91-4a69-adb1-f01b57002d5d-kube-api-access-ql7rp\") pod \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") "
Sep 16 12:57:58.342572 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.342430    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdgrj\" (UniqueName: \"kubernetes.io/projected/805418a0-de25-4fa3-ae83-434958cc0b5f-kube-api-access-gdgrj\") pod \"805418a0-de25-4fa3-ae83-434958cc0b5f\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") "
Sep 16 12:57:58.342572 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.342468    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b96a68e-1b91-4a69-adb1-f01b57002d5d-plugins-config-volume\") pod \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\" (UID: \"7b96a68e-1b91-4a69-adb1-f01b57002d5d\") "
Sep 16 12:57:58.342572 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.342505    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805418a0-de25-4fa3-ae83-434958cc0b5f-plugins-config-volume\") pod \"805418a0-de25-4fa3-ae83-434958cc0b5f\" (UID: \"805418a0-de25-4fa3-ae83-434958cc0b5f\") "
Sep 16 12:57:58.342900 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.342874    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b96a68e-1b91-4a69-adb1-f01b57002d5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b96a68e-1b91-4a69-adb1-f01b57002d5d" (UID: "7b96a68e-1b91-4a69-adb1-f01b57002d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:58.342900 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.342890    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805418a0-de25-4fa3-ae83-434958cc0b5f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "805418a0-de25-4fa3-ae83-434958cc0b5f" (UID: "805418a0-de25-4fa3-ae83-434958cc0b5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:57:58.344729 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.344705    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b96a68e-1b91-4a69-adb1-f01b57002d5d-kube-api-access-ql7rp" (OuterVolumeSpecName: "kube-api-access-ql7rp") pod "7b96a68e-1b91-4a69-adb1-f01b57002d5d" (UID: "7b96a68e-1b91-4a69-adb1-f01b57002d5d"). InnerVolumeSpecName "kube-api-access-ql7rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:58.344804 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.344710    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805418a0-de25-4fa3-ae83-434958cc0b5f-kube-api-access-gdgrj" (OuterVolumeSpecName: "kube-api-access-gdgrj") pod "805418a0-de25-4fa3-ae83-434958cc0b5f" (UID: "805418a0-de25-4fa3-ae83-434958cc0b5f"). InnerVolumeSpecName "kube-api-access-gdgrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:57:58.443558 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.443518    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7rp\" (UniqueName: \"kubernetes.io/projected/7b96a68e-1b91-4a69-adb1-f01b57002d5d-kube-api-access-ql7rp\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:58.443558 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.443550    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdgrj\" (UniqueName: \"kubernetes.io/projected/805418a0-de25-4fa3-ae83-434958cc0b5f-kube-api-access-gdgrj\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:58.443558 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.443562    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b96a68e-1b91-4a69-adb1-f01b57002d5d-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:58.443790 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.443574    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/805418a0-de25-4fa3-ae83-434958cc0b5f-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 12:57:58.685774 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.685736    2559 generic.go:356] "Generic (PLEG): container finished" podID="7b96a68e-1b91-4a69-adb1-f01b57002d5d" containerID="2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37" exitCode=0
Sep 16 12:57:58.685971 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.685793    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"
Sep 16 12:57:58.685971 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.685793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v" event={"ID":"7b96a68e-1b91-4a69-adb1-f01b57002d5d","Type":"ContainerDied","Data":"2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37"}
Sep 16 12:57:58.685971 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.685856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v" event={"ID":"7b96a68e-1b91-4a69-adb1-f01b57002d5d","Type":"ContainerDied","Data":"e292faebfeaf647afdc55be761195bc3cf2b5ac46323171b4854a7e9ccf37bbe"}
Sep 16 12:57:58.685971 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.685883    2559 scope.go:117] "RemoveContainer" containerID="2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37"
Sep 16 12:57:58.687208 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.687185    2559 generic.go:356] "Generic (PLEG): container finished" podID="805418a0-de25-4fa3-ae83-434958cc0b5f" containerID="3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5" exitCode=0
Sep 16 12:57:58.687313 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.687231    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7" event={"ID":"805418a0-de25-4fa3-ae83-434958cc0b5f","Type":"ContainerDied","Data":"3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5"}
Sep 16 12:57:58.687313 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.687269    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"
Sep 16 12:57:58.687463 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.687272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7" event={"ID":"805418a0-de25-4fa3-ae83-434958cc0b5f","Type":"ContainerDied","Data":"33d67f9c03a2efcbd2ea8e129bc77a7cd732556d25fd0c92be85887cf5922ae9"}
Sep 16 12:57:58.702343 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.702292    2559 scope.go:117] "RemoveContainer" containerID="2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37"
Sep 16 12:57:58.702632 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:57:58.702607    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37\": container with ID starting with 2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37 not found: ID does not exist" containerID="2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37"
Sep 16 12:57:58.702729 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.702641    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37"} err="failed to get container status \"2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37\": rpc error: code = NotFound desc = could not find container \"2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37\": container with ID starting with 2e5dd62e41ca9624d38838435a42a2f70b031392d97ca9ab1ae426931b6f4c37 not found: ID does not exist"
Sep 16 12:57:58.702729 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.702664    2559 scope.go:117] "RemoveContainer" containerID="3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5"
Sep 16 12:57:58.710623 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.710599    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"]
Sep 16 12:57:58.711973 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.711950    2559 scope.go:117] "RemoveContainer" containerID="3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5"
Sep 16 12:57:58.712422 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:57:58.712400    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5\": container with ID starting with 3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5 not found: ID does not exist" containerID="3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5"
Sep 16 12:57:58.712599 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.712571    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5"} err="failed to get container status \"3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5\": rpc error: code = NotFound desc = could not find container \"3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5\": container with ID starting with 3368fb6b9c2300bf9b1c10e8e4e6d1c64fb6140469f6ab5a835404fa23e951a5 not found: ID does not exist"
Sep 16 12:57:58.714592 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.714567    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a24d8b-c6b7f8794-vbw9v"]
Sep 16 12:57:58.724296 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.724274    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"]
Sep 16 12:57:58.727950 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:57:58.727926    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a24d8b-7bff5d6c55-2srl7"]
Sep 16 12:58:00.412910 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:00.412882    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b96a68e-1b91-4a69-adb1-f01b57002d5d" path="/var/lib/kubelet/pods/7b96a68e-1b91-4a69-adb1-f01b57002d5d/volumes"
Sep 16 12:58:00.413277 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:00.413227    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="805418a0-de25-4fa3-ae83-434958cc0b5f" path="/var/lib/kubelet/pods/805418a0-de25-4fa3-ae83-434958cc0b5f/volumes"
Sep 16 12:58:10.770137 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770105    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"]
Sep 16 12:58:10.770544 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770501    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="805418a0-de25-4fa3-ae83-434958cc0b5f" containerName="payload-processing"
Sep 16 12:58:10.770544 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770512    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="805418a0-de25-4fa3-ae83-434958cc0b5f" containerName="payload-processing"
Sep 16 12:58:10.770544 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770522    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b96a68e-1b91-4a69-adb1-f01b57002d5d" containerName="payload-pre-processing"
Sep 16 12:58:10.770544 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770527    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b96a68e-1b91-4a69-adb1-f01b57002d5d" containerName="payload-pre-processing"
Sep 16 12:58:10.770544 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770544    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36553dbf-5c97-4996-808a-135fa40a79d5" containerName="maas-api"
Sep 16 12:58:10.770721 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770550    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36553dbf-5c97-4996-808a-135fa40a79d5" containerName="maas-api"
Sep 16 12:58:10.770721 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770607    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="805418a0-de25-4fa3-ae83-434958cc0b5f" containerName="payload-processing"
Sep 16 12:58:10.770721 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770615    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36553dbf-5c97-4996-808a-135fa40a79d5" containerName="maas-api"
Sep 16 12:58:10.770721 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.770622    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b96a68e-1b91-4a69-adb1-f01b57002d5d" containerName="payload-pre-processing"
Sep 16 12:58:10.775214 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.775196    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.779133 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.779108    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"kube-root-ca.crt\""
Sep 16 12:58:10.779575 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.779557    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"default-dockercfg-vb4tt\""
Sep 16 12:58:10.779818 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.779803    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"openshift-service-ca.crt\""
Sep 16 12:58:10.782352 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.782314    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"premium-w0-490797-kserve-self-signed-certs\""
Sep 16 12:58:10.794716 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.794692    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"]
Sep 16 12:58:10.859793 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.859756    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m8bn\" (UniqueName: \"kubernetes.io/projected/092aa52a-34a3-448d-8a29-ba6d7730c70b-kube-api-access-8m8bn\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.859944 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.859821    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092aa52a-34a3-448d-8a29-ba6d7730c70b-tls-certs\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.859944 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.859880    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-home\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.859944 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.859927    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-model-cache\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.860588 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.860571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-tmp-dir\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.860652 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.860604    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-dshm\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.961658 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.961613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-home\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.961658 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.961663    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-model-cache\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.961909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.961720    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-tmp-dir\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.961909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.961748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-dshm\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.961909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.961809    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m8bn\" (UniqueName: \"kubernetes.io/projected/092aa52a-34a3-448d-8a29-ba6d7730c70b-kube-api-access-8m8bn\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.961909 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.961844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092aa52a-34a3-448d-8a29-ba6d7730c70b-tls-certs\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.962101 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.962072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-home\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.962149 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.962132    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-tmp-dir\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.962194 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.962157    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-model-cache\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.964222 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.964195    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-dshm\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.964508 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.964490    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092aa52a-34a3-448d-8a29-ba6d7730c70b-tls-certs\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:10.994421 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:10.994389    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m8bn\" (UniqueName: \"kubernetes.io/projected/092aa52a-34a3-448d-8a29-ba6d7730c70b-kube-api-access-8m8bn\") pod \"premium-w0-490797-kserve-86cb4454b-n52g9\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") " pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:11.084715 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:11.084617    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:11.380657 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:11.380625    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"]
Sep 16 12:58:11.381672 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:58:11.381644    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod092aa52a_34a3_448d_8a29_ba6d7730c70b.slice/crio-92e6ddf8f6006e2062189f10357a9d1c5e71f451f0b40c54d1c067fc3a24d254 WatchSource:0}: Error finding container 92e6ddf8f6006e2062189f10357a9d1c5e71f451f0b40c54d1c067fc3a24d254: Status 404 returned error can't find the container with id 92e6ddf8f6006e2062189f10357a9d1c5e71f451f0b40c54d1c067fc3a24d254
Sep 16 12:58:11.755737 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:11.755695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9" event={"ID":"092aa52a-34a3-448d-8a29-ba6d7730c70b","Type":"ContainerStarted","Data":"0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab"}
Sep 16 12:58:11.755917 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:11.755746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9" event={"ID":"092aa52a-34a3-448d-8a29-ba6d7730c70b","Type":"ContainerStarted","Data":"92e6ddf8f6006e2062189f10357a9d1c5e71f451f0b40c54d1c067fc3a24d254"}
Sep 16 12:58:11.777062 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:11.777014    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9" podStartSLOduration=1.7769979550000001 podStartE2EDuration="1.776997955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:11.776498919 +0000 UTC m=+1193.859941948" watchObservedRunningTime="2026-09-16 12:58:11.776997955 +0000 UTC m=+1193.860441001"
Sep 16 12:58:15.877516 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:15.877484    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:58:18.316741 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.316713    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"]
Sep 16 12:58:18.320232 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.320209    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.323213 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.323186    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nm49v\""
Sep 16 12:58:18.323350 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.323231    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:58:18.323907 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.323890    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7087d2-serving-cert\""
Sep 16 12:58:18.324118 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.324101    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:58:18.324385 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.324368    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:58:18.332483 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.332459    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"]
Sep 16 12:58:18.434576 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.434544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e68b007-e840-4933-bf23-2cadf1d5d1e9-openshift-service-ca\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.434771 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.434591    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd642\" (UniqueName: \"kubernetes.io/projected/5e68b007-e840-4933-bf23-2cadf1d5d1e9-kube-api-access-xd642\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.434771 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.434658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-metrics-certs\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.434771 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.434721    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-maas-api-tls\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.535904 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.535877    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-maas-api-tls\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.536008 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.535965    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e68b007-e840-4933-bf23-2cadf1d5d1e9-openshift-service-ca\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.536050 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.536008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd642\" (UniqueName: \"kubernetes.io/projected/5e68b007-e840-4933-bf23-2cadf1d5d1e9-kube-api-access-xd642\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.536105 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.536051    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-metrics-certs\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.536661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.536597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e68b007-e840-4933-bf23-2cadf1d5d1e9-openshift-service-ca\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.538532 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.538502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-maas-api-tls\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.538649 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.538586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-metrics-certs\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.546306 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.546284    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd642\" (UniqueName: \"kubernetes.io/projected/5e68b007-e840-4933-bf23-2cadf1d5d1e9-kube-api-access-xd642\") pod \"maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.631308 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.631231    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:18.907034 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.907001    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"]
Sep 16 12:58:18.909044 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:58:18.909016    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e68b007_e840_4933_bf23_2cadf1d5d1e9.slice/crio-9e8b29cdd05af014e658aa5ad6ac402198b7ad40e208cebe28501140c55132b6 WatchSource:0}: Error finding container 9e8b29cdd05af014e658aa5ad6ac402198b7ad40e208cebe28501140c55132b6: Status 404 returned error can't find the container with id 9e8b29cdd05af014e658aa5ad6ac402198b7ad40e208cebe28501140c55132b6
Sep 16 12:58:18.910961 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:18.910938    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:58:19.143248 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.143216    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"]
Sep 16 12:58:19.147881 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.147865    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.150139 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.150118    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-7087d2\""
Sep 16 12:58:19.150204 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.150188    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-7087d2-dockercfg-z4hst\""
Sep 16 12:58:19.157027 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.156969    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"]
Sep 16 12:58:19.243005 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.242964    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a373aa-ed83-4e02-9f03-08472f951319-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.243178 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.243134    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb658\" (UniqueName: \"kubernetes.io/projected/c1a373aa-ed83-4e02-9f03-08472f951319-kube-api-access-hb658\") pod \"payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.343757 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.343680    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb658\" (UniqueName: \"kubernetes.io/projected/c1a373aa-ed83-4e02-9f03-08472f951319-kube-api-access-hb658\") pod \"payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.343757 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.343721    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a373aa-ed83-4e02-9f03-08472f951319-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.344454 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.344436    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a373aa-ed83-4e02-9f03-08472f951319-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.351955 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.351934    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb658\" (UniqueName: \"kubernetes.io/projected/c1a373aa-ed83-4e02-9f03-08472f951319-kube-api-access-hb658\") pod \"payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.459329 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.459299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:19.731481 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.731301    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"]
Sep 16 12:58:19.733981 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:58:19.733953    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a373aa_ed83_4e02_9f03_08472f951319.slice/crio-f182fc454e0d6fa6ddaaf256b841da8539da395d8de829b545e586bdc9ae6375 WatchSource:0}: Error finding container f182fc454e0d6fa6ddaaf256b841da8539da395d8de829b545e586bdc9ae6375: Status 404 returned error can't find the container with id f182fc454e0d6fa6ddaaf256b841da8539da395d8de829b545e586bdc9ae6375
Sep 16 12:58:19.800625 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.800586    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c" event={"ID":"c1a373aa-ed83-4e02-9f03-08472f951319","Type":"ContainerStarted","Data":"f182fc454e0d6fa6ddaaf256b841da8539da395d8de829b545e586bdc9ae6375"}
Sep 16 12:58:19.802080 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.802052    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw" event={"ID":"5e68b007-e840-4933-bf23-2cadf1d5d1e9","Type":"ContainerStarted","Data":"595d31608863e1208da209fd3b382297339ad6bbe3a9dc00d04413547eae3091"}
Sep 16 12:58:19.802162 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.802099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw" event={"ID":"5e68b007-e840-4933-bf23-2cadf1d5d1e9","Type":"ContainerStarted","Data":"9e8b29cdd05af014e658aa5ad6ac402198b7ad40e208cebe28501140c55132b6"}
Sep 16 12:58:19.802162 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.802149    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:19.822193 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:19.822136    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw" podStartSLOduration=1.4425920620000001 podStartE2EDuration="1.822120351s" totalImagesPullingTime="379.528289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:19.820418427 +0000 UTC m=+1201.903861466" watchObservedRunningTime="2026-09-16 12:58:19.822120351 +0000 UTC m=+1201.905563407"
Sep 16 12:58:20.809612 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:20.809575    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c" event={"ID":"c1a373aa-ed83-4e02-9f03-08472f951319","Type":"ContainerStarted","Data":"3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d"}
Sep 16 12:58:20.810095 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:20.809826    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:20.826573 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:20.826510    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c" podStartSLOduration=1.8264914920000002 podStartE2EDuration="1.826491492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:58:20.825979397 +0000 UTC m=+1202.909422431" watchObservedRunningTime="2026-09-16 12:58:20.826491492 +0000 UTC m=+1202.909934506"
Sep 16 12:58:21.085040 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:21.084907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:21.085040 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:21.085012    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:21.103844 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:21.103815    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:21.826869 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:21.826838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 12:58:25.815028 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:25.815002    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 12:58:31.817283 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:31.817255    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 12:58:41.160166 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:58:41.160131    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:59:00.553205 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:00.553174    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:59:20.110074 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.109988    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"]
Sep 16 12:59:20.114149 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.114127    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.116406 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.116386    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e937b400-serving-cert\""
Sep 16 12:59:20.125189 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.125170    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"]
Sep 16 12:59:20.145989 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.145961    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-metrics-certs\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.146113 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.146010    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.146113 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.146080    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-openshift-service-ca\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.146187 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.146120    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94zfw\" (UniqueName: \"kubernetes.io/projected/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-kube-api-access-94zfw\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.247048 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.247013    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.247203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.247055    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-openshift-service-ca\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.247203 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.247087    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94zfw\" (UniqueName: \"kubernetes.io/projected/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-kube-api-access-94zfw\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.247203 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:59:20.247176    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-serving-cert: secret "maas-api-e2e-mt-e937b400-serving-cert" not found
Sep 16 12:59:20.247310 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.247229    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-metrics-certs\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.247310 ip-10-0-143-72 kubenswrapper[2559]: E0916 12:59:20.247244    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls podName:bc7d830d-0b45-4e70-a89b-1e96ffe36d9f nodeName:}" failed. No retries permitted until 2026-09-16 12:59:20.747223191 +0000 UTC m=+1262.830666187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls") pod "maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" (UID: "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f") : secret "maas-api-e2e-mt-e937b400-serving-cert" not found
Sep 16 12:59:20.247717 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.247700    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-openshift-service-ca\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.249661 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.249645    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-metrics-certs\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.255798 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.255773    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94zfw\" (UniqueName: \"kubernetes.io/projected/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-kube-api-access-94zfw\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.744039 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.744004    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"]
Sep 16 12:59:20.747689 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.747668    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:20.750350 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.750309    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e937b400\""
Sep 16 12:59:20.750453 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.750365    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e937b400-dockercfg-bj8js\""
Sep 16 12:59:20.752349 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.752311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.754946 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.754899    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls\") pod \"maas-api-e2e-mt-e937b400-7985cc5bc5-bft24\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:20.759556 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.759533    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"]
Sep 16 12:59:20.853406 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.853377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:20.853573 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.853417    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvh8\" (UniqueName: \"kubernetes.io/projected/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-kube-api-access-4gvh8\") pod \"payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:20.954349 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.954288    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:20.954512 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.954368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvh8\" (UniqueName: \"kubernetes.io/projected/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-kube-api-access-4gvh8\") pod \"payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:20.954999 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.954979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:20.962524 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:20.962501    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvh8\" (UniqueName: \"kubernetes.io/projected/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-kube-api-access-4gvh8\") pod \"payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:21.025508 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:21.025447    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:21.067782 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:21.067750    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:21.297032 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:21.297010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"]
Sep 16 12:59:21.298973 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:59:21.298948    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7d830d_0b45_4e70_a89b_1e96ffe36d9f.slice/crio-fb6b848ad2112052a942bc2b9f7feb785fc775be5a981487dcb970346e576157 WatchSource:0}: Error finding container fb6b848ad2112052a942bc2b9f7feb785fc775be5a981487dcb970346e576157: Status 404 returned error can't find the container with id fb6b848ad2112052a942bc2b9f7feb785fc775be5a981487dcb970346e576157
Sep 16 12:59:21.352658 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:21.352611    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"]
Sep 16 12:59:21.354410 ip-10-0-143-72 kubenswrapper[2559]: W0916 12:59:21.354374    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cbb562_d860_43b1_b9cc_8e2efcdcd8f8.slice/crio-41bcd9789215c2ab964ca75c06ae4b9db050e70cef445643fee2a53ae496c0f7 WatchSource:0}: Error finding container 41bcd9789215c2ab964ca75c06ae4b9db050e70cef445643fee2a53ae496c0f7: Status 404 returned error can't find the container with id 41bcd9789215c2ab964ca75c06ae4b9db050e70cef445643fee2a53ae496c0f7
Sep 16 12:59:22.118865 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.118775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" event={"ID":"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f","Type":"ContainerStarted","Data":"adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998"}
Sep 16 12:59:22.118865 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.118841    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:22.118865 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.118859    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" event={"ID":"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f","Type":"ContainerStarted","Data":"fb6b848ad2112052a942bc2b9f7feb785fc775be5a981487dcb970346e576157"}
Sep 16 12:59:22.120426 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.120387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6" event={"ID":"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8","Type":"ContainerStarted","Data":"7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c"}
Sep 16 12:59:22.120426 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.120427    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6" event={"ID":"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8","Type":"ContainerStarted","Data":"41bcd9789215c2ab964ca75c06ae4b9db050e70cef445643fee2a53ae496c0f7"}
Sep 16 12:59:22.120623 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.120446    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 12:59:22.142270 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.142230    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" podStartSLOduration=1.745113864 podStartE2EDuration="2.142216265s" totalImagesPullingTime="397.102401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:22.140554395 +0000 UTC m=+1264.223997435" watchObservedRunningTime="2026-09-16 12:59:22.142216265 +0000 UTC m=+1264.225659279"
Sep 16 12:59:22.159354 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:22.159294    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6" podStartSLOduration=2.159282308 podStartE2EDuration="2.159282308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:59:22.157739906 +0000 UTC m=+1264.241182931" watchObservedRunningTime="2026-09-16 12:59:22.159282308 +0000 UTC m=+1264.242725321"
Sep 16 12:59:28.132497 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:28.132469    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 12:59:33.053490 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:33.053452    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 12:59:33.128179 ip-10-0-143-72 kubenswrapper[2559]: I0916 12:59:33.128140    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 13:00:00.157238 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.157207    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"]
Sep 16 13:00:00.162977 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.162954    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"
Sep 16 13:00:00.165682 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.165659    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:00:00.174836 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.174815    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"]
Sep 16 13:00:00.183722 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.183702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"
Sep 16 13:00:00.188385 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.188359    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"]
Sep 16 13:00:00.192841 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.192803    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"]
Sep 16 13:00:00.192964 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.192930    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"
Sep 16 13:00:00.197096 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.197072    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"
Sep 16 13:00:00.225534 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.225497    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"]
Sep 16 13:00:00.227390 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.227363    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"]
Sep 16 13:00:00.235867 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.235841    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"
Sep 16 13:00:00.244366 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.244344    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"
Sep 16 13:00:00.473005 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.472980    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"]
Sep 16 13:00:00.474863 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:00.474834    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1be646d_4422_42e8_91b3_0b9aa3f4685b.slice/crio-8cba73e68bda1d876c23746c7deb297c32e8c514e1b0756718381bd6d258c0e6 WatchSource:0}: Error finding container 8cba73e68bda1d876c23746c7deb297c32e8c514e1b0756718381bd6d258c0e6: Status 404 returned error can't find the container with id 8cba73e68bda1d876c23746c7deb297c32e8c514e1b0756718381bd6d258c0e6
Sep 16 13:00:00.524645 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.524621    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"]
Sep 16 13:00:00.526425 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:00.526396    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cffb452_9cd6_446b_a1cb_5c5ffca11f8d.slice/crio-1c002e62c15d78952e052fe946ec4f7b8e1d67e0f572765709278a3622be4e8b WatchSource:0}: Error finding container 1c002e62c15d78952e052fe946ec4f7b8e1d67e0f572765709278a3622be4e8b: Status 404 returned error can't find the container with id 1c002e62c15d78952e052fe946ec4f7b8e1d67e0f572765709278a3622be4e8b
Sep 16 13:00:00.535842 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:00.535821    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"]
Sep 16 13:00:00.538204 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:00.538180    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1131c3d_0451_43c2_8948_c7da878ab80a.slice/crio-440d6c20cbb7e4ddd986f76210fcc963e17a4b45912af1d74c00eab9e606cdd9 WatchSource:0}: Error finding container 440d6c20cbb7e4ddd986f76210fcc963e17a4b45912af1d74c00eab9e606cdd9: Status 404 returned error can't find the container with id 440d6c20cbb7e4ddd986f76210fcc963e17a4b45912af1d74c00eab9e606cdd9
Sep 16 13:00:01.339568 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:01.339534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47" event={"ID":"f1be646d-4422-42e8-91b3-0b9aa3f4685b","Type":"ContainerStarted","Data":"8cba73e68bda1d876c23746c7deb297c32e8c514e1b0756718381bd6d258c0e6"}
Sep 16 13:00:01.340771 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:01.340746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2" event={"ID":"f1131c3d-0451-43c2-8948-c7da878ab80a","Type":"ContainerStarted","Data":"440d6c20cbb7e4ddd986f76210fcc963e17a4b45912af1d74c00eab9e606cdd9"}
Sep 16 13:00:01.342000 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:01.341982    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g" event={"ID":"8cffb452-9cd6-446b-a1cb-5c5ffca11f8d","Type":"ContainerStarted","Data":"1c002e62c15d78952e052fe946ec4f7b8e1d67e0f572765709278a3622be4e8b"}
Sep 16 13:00:04.361848 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.361816    2559 generic.go:356] "Generic (PLEG): container finished" podID="f1be646d-4422-42e8-91b3-0b9aa3f4685b" containerID="6d92efbdbcf7c4e65ef3ea4da22b3ba93e3e8e1a4ad8d7227f73a5e4d82b5256" exitCode=0
Sep 16 13:00:04.362292 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.361876    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47" event={"ID":"f1be646d-4422-42e8-91b3-0b9aa3f4685b","Type":"ContainerDied","Data":"6d92efbdbcf7c4e65ef3ea4da22b3ba93e3e8e1a4ad8d7227f73a5e4d82b5256"}
Sep 16 13:00:04.363496 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.363463    2559 generic.go:356] "Generic (PLEG): container finished" podID="8cffb452-9cd6-446b-a1cb-5c5ffca11f8d" containerID="d786fa03a45324a13f6a6b7a66e06715e366091c00f54c81ad913be234c8feb4" exitCode=0
Sep 16 13:00:04.363631 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.363511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g" event={"ID":"8cffb452-9cd6-446b-a1cb-5c5ffca11f8d","Type":"ContainerDied","Data":"d786fa03a45324a13f6a6b7a66e06715e366091c00f54c81ad913be234c8feb4"}
Sep 16 13:00:04.618956 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.618862    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"]
Sep 16 13:00:04.619127 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.619099    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" podUID="bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" containerName="maas-api" containerID="cri-o://adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998" gracePeriod=30
Sep 16 13:00:04.688019 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.687983    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"]
Sep 16 13:00:04.688291 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.688243    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6" podUID="27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" containerName="payload-pre-processing" containerID="cri-o://7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c" gracePeriod=30
Sep 16 13:00:04.943287 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.943261    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 13:00:04.985794 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:04.985767    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 13:00:05.023950 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.023919    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls\") pod \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") "
Sep 16 13:00:05.024108 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.023975    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94zfw\" (UniqueName: \"kubernetes.io/projected/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-kube-api-access-94zfw\") pod \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") "
Sep 16 13:00:05.024108 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.024071    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-openshift-service-ca\") pod \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") "
Sep 16 13:00:05.024228 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.024136    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-metrics-certs\") pod \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\" (UID: \"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f\") "
Sep 16 13:00:05.024508 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.024474    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" (UID: "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:05.026469 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.026442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" (UID: "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:00:05.026579 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.026502    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-kube-api-access-94zfw" (OuterVolumeSpecName: "kube-api-access-94zfw") pod "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" (UID: "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f"). InnerVolumeSpecName "kube-api-access-94zfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:05.026691 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.026674    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" (UID: "bc7d830d-0b45-4e70-a89b-1e96ffe36d9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:00:05.125849 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.125803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gvh8\" (UniqueName: \"kubernetes.io/projected/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-kube-api-access-4gvh8\") pod \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") "
Sep 16 13:00:05.125849 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.125856    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-plugins-config-volume\") pod \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\" (UID: \"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8\") "
Sep 16 13:00:05.126207 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.126187    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.126259 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.126213    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.126259 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.126230    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.126259 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.126243    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94zfw\" (UniqueName: \"kubernetes.io/projected/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f-kube-api-access-94zfw\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.126386 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.126262    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" (UID: "27cbb562-d860-43b1-b9cc-8e2efcdcd8f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:05.128116 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.128094    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-kube-api-access-4gvh8" (OuterVolumeSpecName: "kube-api-access-4gvh8") pod "27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" (UID: "27cbb562-d860-43b1-b9cc-8e2efcdcd8f8"). InnerVolumeSpecName "kube-api-access-4gvh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:05.227382 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.227314    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvh8\" (UniqueName: \"kubernetes.io/projected/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-kube-api-access-4gvh8\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.227382 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.227373    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:05.370418 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.370278    2559 generic.go:356] "Generic (PLEG): container finished" podID="27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" containerID="7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c" exitCode=0
Sep 16 13:00:05.370418 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.370370    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"
Sep 16 13:00:05.371020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.370451    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6" event={"ID":"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8","Type":"ContainerDied","Data":"7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c"}
Sep 16 13:00:05.371020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.370551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6" event={"ID":"27cbb562-d860-43b1-b9cc-8e2efcdcd8f8","Type":"ContainerDied","Data":"41bcd9789215c2ab964ca75c06ae4b9db050e70cef445643fee2a53ae496c0f7"}
Sep 16 13:00:05.371020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.370581    2559 scope.go:117] "RemoveContainer" containerID="7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c"
Sep 16 13:00:05.371892 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.371872    2559 generic.go:356] "Generic (PLEG): container finished" podID="bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" containerID="adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998" exitCode=0
Sep 16 13:00:05.371983 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.371939    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"
Sep 16 13:00:05.371983 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.371961    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" event={"ID":"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f","Type":"ContainerDied","Data":"adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998"}
Sep 16 13:00:05.372084 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.372010    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24" event={"ID":"bc7d830d-0b45-4e70-a89b-1e96ffe36d9f","Type":"ContainerDied","Data":"fb6b848ad2112052a942bc2b9f7feb785fc775be5a981487dcb970346e576157"}
Sep 16 13:00:05.382924 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.382895    2559 scope.go:117] "RemoveContainer" containerID="7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c"
Sep 16 13:00:05.383541 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:00:05.383508    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c\": container with ID starting with 7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c not found: ID does not exist" containerID="7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c"
Sep 16 13:00:05.383648 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.383555    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c"} err="failed to get container status \"7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c\": rpc error: code = NotFound desc = could not find container \"7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c\": container with ID starting with 7397f43eb436396162270c12deaca6930d43c44e1b04ae0daa2a91a61dea4c6c not found: ID does not exist"
Sep 16 13:00:05.383648 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.383582    2559 scope.go:117] "RemoveContainer" containerID="adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998"
Sep 16 13:00:05.395409 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.395390    2559 scope.go:117] "RemoveContainer" containerID="adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998"
Sep 16 13:00:05.395879 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:00:05.395849    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998\": container with ID starting with adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998 not found: ID does not exist" containerID="adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998"
Sep 16 13:00:05.396003 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.395887    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998"} err="failed to get container status \"adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998\": rpc error: code = NotFound desc = could not find container \"adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998\": container with ID starting with adda62a4f27af79c6df11e7088f443bf9ce61976b4dccf78dadebc710612d998 not found: ID does not exist"
Sep 16 13:00:05.400174 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.400139    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"]
Sep 16 13:00:05.405139 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.405112    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e937b400-748585c95d-9mwr6"]
Sep 16 13:00:05.421211 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.421179    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"]
Sep 16 13:00:05.426198 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.426174    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e937b400-7985cc5bc5-bft24"]
Sep 16 13:00:05.499831 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.499804    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"
Sep 16 13:00:05.531836 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:05.531814    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"
Sep 16 13:00:06.380440 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.380408    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"
Sep 16 13:00:06.380864 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.380410    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g" event={"ID":"8cffb452-9cd6-446b-a1cb-5c5ffca11f8d","Type":"ContainerDied","Data":"1c002e62c15d78952e052fe946ec4f7b8e1d67e0f572765709278a3622be4e8b"}
Sep 16 13:00:06.380864 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.380555    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c002e62c15d78952e052fe946ec4f7b8e1d67e0f572765709278a3622be4e8b"
Sep 16 13:00:06.381798 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.381775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47" event={"ID":"f1be646d-4422-42e8-91b3-0b9aa3f4685b","Type":"ContainerDied","Data":"8cba73e68bda1d876c23746c7deb297c32e8c514e1b0756718381bd6d258c0e6"}
Sep 16 13:00:06.381903 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.381819    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cba73e68bda1d876c23746c7deb297c32e8c514e1b0756718381bd6d258c0e6"
Sep 16 13:00:06.381903 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.381784    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"
Sep 16 13:00:06.412809 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.412783    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" path="/var/lib/kubelet/pods/27cbb562-d860-43b1-b9cc-8e2efcdcd8f8/volumes"
Sep 16 13:00:06.413196 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:06.413183    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" path="/var/lib/kubelet/pods/bc7d830d-0b45-4e70-a89b-1e96ffe36d9f/volumes"
Sep 16 13:00:07.453356 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:07.453300    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:00:09.398975 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:09.398880    2559 generic.go:356] "Generic (PLEG): container finished" podID="f1131c3d-0451-43c2-8948-c7da878ab80a" containerID="8168a34c50f64ad26ff9adc2d9dbfa76b2a42cc0f1930f88afe7effed0536dfa" exitCode=0
Sep 16 13:00:09.398975 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:09.398941    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2" event={"ID":"f1131c3d-0451-43c2-8948-c7da878ab80a","Type":"ContainerDied","Data":"8168a34c50f64ad26ff9adc2d9dbfa76b2a42cc0f1930f88afe7effed0536dfa"}
Sep 16 13:00:10.546138 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:10.546109    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"
Sep 16 13:00:11.411174 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:11.411143    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2"
Sep 16 13:00:11.411372 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:11.411176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29826060-2ckx2" event={"ID":"f1131c3d-0451-43c2-8948-c7da878ab80a","Type":"ContainerDied","Data":"440d6c20cbb7e4ddd986f76210fcc963e17a4b45912af1d74c00eab9e606cdd9"}
Sep 16 13:00:11.411372 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:11.411222    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="440d6c20cbb7e4ddd986f76210fcc963e17a4b45912af1d74c00eab9e606cdd9"
Sep 16 13:00:16.361635 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.361597    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"]
Sep 16 13:00:16.362196 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362177    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" containerName="maas-api"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362198    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" containerName="maas-api"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362225    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cffb452-9cd6-446b-a1cb-5c5ffca11f8d" containerName="cleanup"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362235    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cffb452-9cd6-446b-a1cb-5c5ffca11f8d" containerName="cleanup"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362243    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1be646d-4422-42e8-91b3-0b9aa3f4685b" containerName="cleanup"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362251    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1be646d-4422-42e8-91b3-0b9aa3f4685b" containerName="cleanup"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362261    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1131c3d-0451-43c2-8948-c7da878ab80a" containerName="cleanup"
Sep 16 13:00:16.362278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362269    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1131c3d-0451-43c2-8948-c7da878ab80a" containerName="cleanup"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362286    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" containerName="payload-pre-processing"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362295    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" containerName="payload-pre-processing"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362423    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc7d830d-0b45-4e70-a89b-1e96ffe36d9f" containerName="maas-api"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362436    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cffb452-9cd6-446b-a1cb-5c5ffca11f8d" containerName="cleanup"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362444    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cbb562-d860-43b1-b9cc-8e2efcdcd8f8" containerName="payload-pre-processing"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362458    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1131c3d-0451-43c2-8948-c7da878ab80a" containerName="cleanup"
Sep 16 13:00:16.362609 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.362467    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1be646d-4422-42e8-91b3-0b9aa3f4685b" containerName="cleanup"
Sep 16 13:00:16.365597 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.365582    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.367886 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.367863    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"e2e-conflict-a-32338a53-kserve-self-signed-certs\""
Sep 16 13:00:16.374273 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.374250    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"]
Sep 16 13:00:16.541381 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.541343    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tls-certs\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.541566 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.541446    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-dshm\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.541566 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.541516    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-model-cache\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.541566 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.541545    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tmp-dir\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.541691 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.541602    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-home\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.541691 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.541624    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncgz\" (UniqueName: \"kubernetes.io/projected/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-kube-api-access-rncgz\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.642853 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.642763    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-dshm\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.642853 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.642838    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-model-cache\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643038 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.642865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tmp-dir\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643038 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.642915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-home\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643038 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.642944    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncgz\" (UniqueName: \"kubernetes.io/projected/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-kube-api-access-rncgz\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643038 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.643000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tls-certs\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643368 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.643304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-model-cache\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643480 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.643369    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tmp-dir\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.643480 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.643380    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-home\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.645233 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.645212    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-dshm\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.645527 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.645508    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tls-certs\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.651135 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.651115    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncgz\" (UniqueName: \"kubernetes.io/projected/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-kube-api-access-rncgz\") pod \"e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") " pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.675730 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.675694    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:16.948401 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:16.948369    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"]
Sep 16 13:00:16.951950 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:16.951917    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f7f359c_0d64_4544_a92c_b7918f2f8bd0.slice/crio-2b07fe3bf389c9dda40d5b89c52cbd22a2d3acf24e99436793f1d8ee50ed84b1 WatchSource:0}: Error finding container 2b07fe3bf389c9dda40d5b89c52cbd22a2d3acf24e99436793f1d8ee50ed84b1: Status 404 returned error can't find the container with id 2b07fe3bf389c9dda40d5b89c52cbd22a2d3acf24e99436793f1d8ee50ed84b1
Sep 16 13:00:17.440904 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:17.440868    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2" event={"ID":"7f7f359c-0d64-4544-a92c-b7918f2f8bd0","Type":"ContainerStarted","Data":"83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9"}
Sep 16 13:00:17.441266 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:17.440917    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2" event={"ID":"7f7f359c-0d64-4544-a92c-b7918f2f8bd0","Type":"ContainerStarted","Data":"2b07fe3bf389c9dda40d5b89c52cbd22a2d3acf24e99436793f1d8ee50ed84b1"}
Sep 16 13:00:17.458211 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:17.458159    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2" podStartSLOduration=1.458137839 podStartE2EDuration="1.458137839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:17.457921863 +0000 UTC m=+1319.541364913" watchObservedRunningTime="2026-09-16 13:00:17.458137839 +0000 UTC m=+1319.541580853"
Sep 16 13:00:23.654911 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.654875    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"]
Sep 16 13:00:23.660021 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.660000    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.662424 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.662403    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-490797\"/\"distinct-w0-490797-kserve-self-signed-certs\""
Sep 16 13:00:23.669732 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.669710    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"]
Sep 16 13:00:23.710510 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.710481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-home\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.710663 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.710518    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-dshm\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.710663 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.710546    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-tmp-dir\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.710663 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.710567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b253a0c6-3665-4d05-9283-13ec07bfa15b-tls-certs\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.710663 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.710641    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4jrj\" (UniqueName: \"kubernetes.io/projected/b253a0c6-3665-4d05-9283-13ec07bfa15b-kube-api-access-v4jrj\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.710850 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.710769    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-model-cache\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811573 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-model-cache\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811751 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811604    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-home\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811751 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811622    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-dshm\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811751 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811638    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-tmp-dir\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811751 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811652    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b253a0c6-3665-4d05-9283-13ec07bfa15b-tls-certs\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811958 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4jrj\" (UniqueName: \"kubernetes.io/projected/b253a0c6-3665-4d05-9283-13ec07bfa15b-kube-api-access-v4jrj\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.811958 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811945    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-model-cache\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.812056 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.811975    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-tmp-dir\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.812095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.812071    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-home\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.814018 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.813990    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-dshm\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.814286 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.814269    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b253a0c6-3665-4d05-9283-13ec07bfa15b-tls-certs\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.820115 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.820097    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4jrj\" (UniqueName: \"kubernetes.io/projected/b253a0c6-3665-4d05-9283-13ec07bfa15b-kube-api-access-v4jrj\") pod \"distinct-w0-490797-kserve-84869b6957-lqf8z\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") " pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:23.970270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:23.970241    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:24.244070 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:24.244046    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"]
Sep 16 13:00:24.246775 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:24.246749    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb253a0c6_3665_4d05_9283_13ec07bfa15b.slice/crio-37d4e9706334d3a146851a74489c210561daee360d2585b934de891db8c3aecf WatchSource:0}: Error finding container 37d4e9706334d3a146851a74489c210561daee360d2585b934de891db8c3aecf: Status 404 returned error can't find the container with id 37d4e9706334d3a146851a74489c210561daee360d2585b934de891db8c3aecf
Sep 16 13:00:24.476380 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:24.476342    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z" event={"ID":"b253a0c6-3665-4d05-9283-13ec07bfa15b","Type":"ContainerStarted","Data":"30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102"}
Sep 16 13:00:24.476380 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:24.476388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z" event={"ID":"b253a0c6-3665-4d05-9283-13ec07bfa15b","Type":"ContainerStarted","Data":"37d4e9706334d3a146851a74489c210561daee360d2585b934de891db8c3aecf"}
Sep 16 13:00:24.495960 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:24.495859    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z" podStartSLOduration=1.495845714 podStartE2EDuration="1.495845714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:24.494129306 +0000 UTC m=+1326.577572342" watchObservedRunningTime="2026-09-16 13:00:24.495845714 +0000 UTC m=+1326.579288727"
Sep 16 13:00:26.676746 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:26.676710    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:26.677120 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:26.676763    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:26.689698 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:26.689674    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:27.504119 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:27.504091    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:00:33.970660 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:33.970615    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:33.971213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:33.970673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:33.983675 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:33.983643    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:34.543429 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:34.543305    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:00:35.001111 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.001074    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"]
Sep 16 13:00:35.004915 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.004896    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.007476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.007455    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-863d9d-serving-cert\""
Sep 16 13:00:35.022469 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.022440    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"]
Sep 16 13:00:35.127096 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.127054    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabb55e7-a3c3-449b-b00e-0efd9b766be1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.127280 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.127107    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smfs5\" (UniqueName: \"kubernetes.io/projected/aabb55e7-a3c3-449b-b00e-0efd9b766be1-kube-api-access-smfs5\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.127280 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.127252    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.127409 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.127299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.228792 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.228744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.228969 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.228800    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.228969 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.228863    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabb55e7-a3c3-449b-b00e-0efd9b766be1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.228969 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.228908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smfs5\" (UniqueName: \"kubernetes.io/projected/aabb55e7-a3c3-449b-b00e-0efd9b766be1-kube-api-access-smfs5\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.229097 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:00:35.228965    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-serving-cert: secret "maas-api-e2e-shared-a-w2-863d9d-serving-cert" not found
Sep 16 13:00:35.229097 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:00:35.229036    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls podName:aabb55e7-a3c3-449b-b00e-0efd9b766be1 nodeName:}" failed. No retries permitted until 2026-09-16 13:00:35.729018135 +0000 UTC m=+1337.812461127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls") pod "maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" (UID: "aabb55e7-a3c3-449b-b00e-0efd9b766be1") : secret "maas-api-e2e-shared-a-w2-863d9d-serving-cert" not found
Sep 16 13:00:35.229611 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.229594    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabb55e7-a3c3-449b-b00e-0efd9b766be1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.231475 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.231457    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.243604 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.243573    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smfs5\" (UniqueName: \"kubernetes.io/projected/aabb55e7-a3c3-449b-b00e-0efd9b766be1-kube-api-access-smfs5\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.736078 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.736028    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.738683 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.738656    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:35.915706 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:35.915677    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:36.004213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.004134    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"]
Sep 16 13:00:36.008952 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.008932    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.016742 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.016713    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-863d9d-dockercfg-xkfl4\""
Sep 16 13:00:36.017178 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.017121    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-863d9d\""
Sep 16 13:00:36.038540 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.038513    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"]
Sep 16 13:00:36.140598 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.140560    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24918870-b1b7-46a0-b9dc-eb6025c56491-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.140796 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.140670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h5tm\" (UniqueName: \"kubernetes.io/projected/24918870-b1b7-46a0-b9dc-eb6025c56491-kube-api-access-4h5tm\") pod \"payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.201435 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.201409    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"]
Sep 16 13:00:36.203774 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:36.203750    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabb55e7_a3c3_449b_b00e_0efd9b766be1.slice/crio-060219ec6f56e834cf0bc9ed979cae99399a278480da7993031f23758b8d461f WatchSource:0}: Error finding container 060219ec6f56e834cf0bc9ed979cae99399a278480da7993031f23758b8d461f: Status 404 returned error can't find the container with id 060219ec6f56e834cf0bc9ed979cae99399a278480da7993031f23758b8d461f
Sep 16 13:00:36.242236 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.242201    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24918870-b1b7-46a0-b9dc-eb6025c56491-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.242458 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.242297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h5tm\" (UniqueName: \"kubernetes.io/projected/24918870-b1b7-46a0-b9dc-eb6025c56491-kube-api-access-4h5tm\") pod \"payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.243185 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.243158    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24918870-b1b7-46a0-b9dc-eb6025c56491-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.280068 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.279991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h5tm\" (UniqueName: \"kubernetes.io/projected/24918870-b1b7-46a0-b9dc-eb6025c56491-kube-api-access-4h5tm\") pod \"payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.296923 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.296889    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"]
Sep 16 13:00:36.297127 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.297100    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw" podUID="5e68b007-e840-4933-bf23-2cadf1d5d1e9" containerName="maas-api" containerID="cri-o://595d31608863e1208da209fd3b382297339ad6bbe3a9dc00d04413547eae3091" gracePeriod=30
Sep 16 13:00:36.327794 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.327766    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:36.351824 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.351789    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"]
Sep 16 13:00:36.355001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.354978    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-explicit-7087d2-29826060-pl57g"]
Sep 16 13:00:36.419485 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.419161    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cffb452-9cd6-446b-a1cb-5c5ffca11f8d" path="/var/lib/kubelet/pods/8cffb452-9cd6-446b-a1cb-5c5ffca11f8d/volumes"
Sep 16 13:00:36.447868 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.447831    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"]
Sep 16 13:00:36.448137 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.448087    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c" podUID="c1a373aa-ed83-4e02-9f03-08472f951319" containerName="payload-pre-processing" containerID="cri-o://3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d" gracePeriod=30
Sep 16 13:00:36.545586 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.545496    2559 generic.go:356] "Generic (PLEG): container finished" podID="5e68b007-e840-4933-bf23-2cadf1d5d1e9" containerID="595d31608863e1208da209fd3b382297339ad6bbe3a9dc00d04413547eae3091" exitCode=0
Sep 16 13:00:36.545730 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.545572    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw" event={"ID":"5e68b007-e840-4933-bf23-2cadf1d5d1e9","Type":"ContainerDied","Data":"595d31608863e1208da209fd3b382297339ad6bbe3a9dc00d04413547eae3091"}
Sep 16 13:00:36.546928 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.546904    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" event={"ID":"aabb55e7-a3c3-449b-b00e-0efd9b766be1","Type":"ContainerStarted","Data":"060219ec6f56e834cf0bc9ed979cae99399a278480da7993031f23758b8d461f"}
Sep 16 13:00:36.562441 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.562414    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 13:00:36.625397 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.625364    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:00:36.644959 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.644927    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"]
Sep 16 13:00:36.646593 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.646565    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-maas-api-tls\") pod \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") "
Sep 16 13:00:36.646695 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.646607    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e68b007-e840-4933-bf23-2cadf1d5d1e9-openshift-service-ca\") pod \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") "
Sep 16 13:00:36.646695 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.646675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd642\" (UniqueName: \"kubernetes.io/projected/5e68b007-e840-4933-bf23-2cadf1d5d1e9-kube-api-access-xd642\") pod \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") "
Sep 16 13:00:36.646841 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.646818    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-metrics-certs\") pod \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\" (UID: \"5e68b007-e840-4933-bf23-2cadf1d5d1e9\") "
Sep 16 13:00:36.647291 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.647243    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e68b007-e840-4933-bf23-2cadf1d5d1e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e68b007-e840-4933-bf23-2cadf1d5d1e9" (UID: "5e68b007-e840-4933-bf23-2cadf1d5d1e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:36.649955 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.649928    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e68b007-e840-4933-bf23-2cadf1d5d1e9-kube-api-access-xd642" (OuterVolumeSpecName: "kube-api-access-xd642") pod "5e68b007-e840-4933-bf23-2cadf1d5d1e9" (UID: "5e68b007-e840-4933-bf23-2cadf1d5d1e9"). InnerVolumeSpecName "kube-api-access-xd642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:36.650042 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.650009    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e68b007-e840-4933-bf23-2cadf1d5d1e9" (UID: "5e68b007-e840-4933-bf23-2cadf1d5d1e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:00:36.651424 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.651400    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e68b007-e840-4933-bf23-2cadf1d5d1e9" (UID: "5e68b007-e840-4933-bf23-2cadf1d5d1e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:00:36.652716 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:00:36.652650    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24918870_b1b7_46a0_b9dc_eb6025c56491.slice/crio-60c4447947ac7bdc96298f2f6a9f53a0c902a746d4d8233e341a77f206c2059f WatchSource:0}: Error finding container 60c4447947ac7bdc96298f2f6a9f53a0c902a746d4d8233e341a77f206c2059f: Status 404 returned error can't find the container with id 60c4447947ac7bdc96298f2f6a9f53a0c902a746d4d8233e341a77f206c2059f
Sep 16 13:00:36.729688 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.729660    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 13:00:36.748361 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.748310    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.748361 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.748361    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e68b007-e840-4933-bf23-2cadf1d5d1e9-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.748635 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.748375    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e68b007-e840-4933-bf23-2cadf1d5d1e9-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.748635 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.748389    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd642\" (UniqueName: \"kubernetes.io/projected/5e68b007-e840-4933-bf23-2cadf1d5d1e9-kube-api-access-xd642\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.849844 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.849550    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a373aa-ed83-4e02-9f03-08472f951319-plugins-config-volume\") pod \"c1a373aa-ed83-4e02-9f03-08472f951319\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") "
Sep 16 13:00:36.849844 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.849704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb658\" (UniqueName: \"kubernetes.io/projected/c1a373aa-ed83-4e02-9f03-08472f951319-kube-api-access-hb658\") pod \"c1a373aa-ed83-4e02-9f03-08472f951319\" (UID: \"c1a373aa-ed83-4e02-9f03-08472f951319\") "
Sep 16 13:00:36.850287 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.850246    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a373aa-ed83-4e02-9f03-08472f951319-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1a373aa-ed83-4e02-9f03-08472f951319" (UID: "c1a373aa-ed83-4e02-9f03-08472f951319"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:00:36.852358 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.852306    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a373aa-ed83-4e02-9f03-08472f951319-kube-api-access-hb658" (OuterVolumeSpecName: "kube-api-access-hb658") pod "c1a373aa-ed83-4e02-9f03-08472f951319" (UID: "c1a373aa-ed83-4e02-9f03-08472f951319"). InnerVolumeSpecName "kube-api-access-hb658". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:00:36.950696 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.950661    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1a373aa-ed83-4e02-9f03-08472f951319-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:36.950696 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:36.950690    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb658\" (UniqueName: \"kubernetes.io/projected/c1a373aa-ed83-4e02-9f03-08472f951319-kube-api-access-hb658\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:00:37.555030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.554931    2559 generic.go:356] "Generic (PLEG): container finished" podID="c1a373aa-ed83-4e02-9f03-08472f951319" containerID="3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d" exitCode=0
Sep 16 13:00:37.555030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.554977    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"
Sep 16 13:00:37.555030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.555004    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c" event={"ID":"c1a373aa-ed83-4e02-9f03-08472f951319","Type":"ContainerDied","Data":"3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d"}
Sep 16 13:00:37.555588 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.555063    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c" event={"ID":"c1a373aa-ed83-4e02-9f03-08472f951319","Type":"ContainerDied","Data":"f182fc454e0d6fa6ddaaf256b841da8539da395d8de829b545e586bdc9ae6375"}
Sep 16 13:00:37.555588 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.555095    2559 scope.go:117] "RemoveContainer" containerID="3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d"
Sep 16 13:00:37.557085 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.557060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh" event={"ID":"24918870-b1b7-46a0-b9dc-eb6025c56491","Type":"ContainerStarted","Data":"8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1"}
Sep 16 13:00:37.557210 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.557124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh" event={"ID":"24918870-b1b7-46a0-b9dc-eb6025c56491","Type":"ContainerStarted","Data":"60c4447947ac7bdc96298f2f6a9f53a0c902a746d4d8233e341a77f206c2059f"}
Sep 16 13:00:37.557210 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.557175    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:00:37.558636 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.558618    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"
Sep 16 13:00:37.558636 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.558638    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw" event={"ID":"5e68b007-e840-4933-bf23-2cadf1d5d1e9","Type":"ContainerDied","Data":"9e8b29cdd05af014e658aa5ad6ac402198b7ad40e208cebe28501140c55132b6"}
Sep 16 13:00:37.561299 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.561277    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" event={"ID":"aabb55e7-a3c3-449b-b00e-0efd9b766be1","Type":"ContainerStarted","Data":"0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce"}
Sep 16 13:00:37.561855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.561838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:37.568635 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.568616    2559 scope.go:117] "RemoveContainer" containerID="3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d"
Sep 16 13:00:37.568971 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:00:37.568952    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d\": container with ID starting with 3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d not found: ID does not exist" containerID="3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d"
Sep 16 13:00:37.569097 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.568979    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d"} err="failed to get container status \"3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d\": rpc error: code = NotFound desc = could not find container \"3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d\": container with ID starting with 3c2af85ad8e2e542f9c4f974759e08b688efcbf270efb4f25b29c8b2c8040e7d not found: ID does not exist"
Sep 16 13:00:37.569097 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.568995    2559 scope.go:117] "RemoveContainer" containerID="595d31608863e1208da209fd3b382297339ad6bbe3a9dc00d04413547eae3091"
Sep 16 13:00:37.581139 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.581097    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh" podStartSLOduration=2.58108102 podStartE2EDuration="2.58108102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:37.580004312 +0000 UTC m=+1339.663447337" watchObservedRunningTime="2026-09-16 13:00:37.58108102 +0000 UTC m=+1339.664524061"
Sep 16 13:00:37.605343 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.605285    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"]
Sep 16 13:00:37.623695 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.623663    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-7087d2-56955d78fd-lml8c"]
Sep 16 13:00:37.644509 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.644422    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" podStartSLOduration=3.247219094 podStartE2EDuration="3.644385401s" totalImagesPullingTime="397.166307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:00:37.637984697 +0000 UTC m=+1339.721427713" watchObservedRunningTime="2026-09-16 13:00:37.644385401 +0000 UTC m=+1339.727828414"
Sep 16 13:00:37.659469 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.659439    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"]
Sep 16 13:00:37.669391 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:37.669360    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7087d2-9676f7bb8-n4lzw"]
Sep 16 13:00:38.419182 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:38.418280    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e68b007-e840-4933-bf23-2cadf1d5d1e9" path="/var/lib/kubelet/pods/5e68b007-e840-4933-bf23-2cadf1d5d1e9/volumes"
Sep 16 13:00:38.419182 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:38.418826    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1a373aa-ed83-4e02-9f03-08472f951319" path="/var/lib/kubelet/pods/c1a373aa-ed83-4e02-9f03-08472f951319/volumes"
Sep 16 13:00:44.580439 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:44.580363    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:00:48.571760 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:00:48.571731    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:01:09.163066 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:09.163035    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:01:10.228742 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.228707    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"]
Sep 16 13:01:10.229343 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.229295    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1a373aa-ed83-4e02-9f03-08472f951319" containerName="payload-pre-processing"
Sep 16 13:01:10.229343 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.229313    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1a373aa-ed83-4e02-9f03-08472f951319" containerName="payload-pre-processing"
Sep 16 13:01:10.229521 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.229346    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e68b007-e840-4933-bf23-2cadf1d5d1e9" containerName="maas-api"
Sep 16 13:01:10.229521 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.229355    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e68b007-e840-4933-bf23-2cadf1d5d1e9" containerName="maas-api"
Sep 16 13:01:10.229521 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.229442    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e68b007-e840-4933-bf23-2cadf1d5d1e9" containerName="maas-api"
Sep 16 13:01:10.229521 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.229459    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1a373aa-ed83-4e02-9f03-08472f951319" containerName="payload-pre-processing"
Sep 16 13:01:10.234160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.234140    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.236720 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.236696    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e5926f-serving-cert\""
Sep 16 13:01:10.249994 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.249964    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"]
Sep 16 13:01:10.389603 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.389567    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z888z\" (UniqueName: \"kubernetes.io/projected/8c7412c7-8864-4562-b334-431a4ad12e97-kube-api-access-z888z\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.389802 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.389607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.389802 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.389663    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.389802 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.389681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7412c7-8864-4562-b334-431a4ad12e97-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.491009 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.490923    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z888z\" (UniqueName: \"kubernetes.io/projected/8c7412c7-8864-4562-b334-431a4ad12e97-kube-api-access-z888z\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.491009 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.490964    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.491009 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.490993    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.491009 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.491011    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7412c7-8864-4562-b334-431a4ad12e97-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.491789 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.491770    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7412c7-8864-4562-b334-431a4ad12e97-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.493582 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.493561    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.493670 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.493660    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.501689 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.501665    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z888z\" (UniqueName: \"kubernetes.io/projected/8c7412c7-8864-4562-b334-431a4ad12e97-kube-api-access-z888z\") pod \"maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.544182 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.544155    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:10.825021 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:10.824999    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"]
Sep 16 13:01:10.827220 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:01:10.827192    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7412c7_8864_4562_b334_431a4ad12e97.slice/crio-b16aef10a1f250eb582f5b50c39531d8ace17c8142648cb0005cf92698726ede WatchSource:0}: Error finding container b16aef10a1f250eb582f5b50c39531d8ace17c8142648cb0005cf92698726ede: Status 404 returned error can't find the container with id b16aef10a1f250eb582f5b50c39531d8ace17c8142648cb0005cf92698726ede
Sep 16 13:01:11.743905 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:11.743866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk" event={"ID":"8c7412c7-8864-4562-b334-431a4ad12e97","Type":"ContainerStarted","Data":"0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652"}
Sep 16 13:01:11.744314 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:11.743920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk" event={"ID":"8c7412c7-8864-4562-b334-431a4ad12e97","Type":"ContainerStarted","Data":"b16aef10a1f250eb582f5b50c39531d8ace17c8142648cb0005cf92698726ede"}
Sep 16 13:01:11.744314 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:11.743973    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:11.767202 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:11.767151    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk" podStartSLOduration=1.296190719 podStartE2EDuration="1.767136043s" totalImagesPullingTime="470.945324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:11.764285141 +0000 UTC m=+1373.847728149" watchObservedRunningTime="2026-09-16 13:01:11.767136043 +0000 UTC m=+1373.850579057"
Sep 16 13:01:16.609374 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:16.609341    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"]
Sep 16 13:01:16.769407 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:16.769345    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45" podUID="36fdc0d0-2fb0-4d96-90e3-09955e545f71" containerName="payload-processing" containerID="cri-o://88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679" gracePeriod=30
Sep 16 13:01:17.029397 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.029369    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 13:01:17.157302 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.157213    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8spw\" (UniqueName: \"kubernetes.io/projected/36fdc0d0-2fb0-4d96-90e3-09955e545f71-kube-api-access-m8spw\") pod \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") "
Sep 16 13:01:17.157493 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.157379    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fdc0d0-2fb0-4d96-90e3-09955e545f71-plugins-config-volume\") pod \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\" (UID: \"36fdc0d0-2fb0-4d96-90e3-09955e545f71\") "
Sep 16 13:01:17.157768 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.157744    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fdc0d0-2fb0-4d96-90e3-09955e545f71-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36fdc0d0-2fb0-4d96-90e3-09955e545f71" (UID: "36fdc0d0-2fb0-4d96-90e3-09955e545f71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:17.159589 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.159569    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fdc0d0-2fb0-4d96-90e3-09955e545f71-kube-api-access-m8spw" (OuterVolumeSpecName: "kube-api-access-m8spw") pod "36fdc0d0-2fb0-4d96-90e3-09955e545f71" (UID: "36fdc0d0-2fb0-4d96-90e3-09955e545f71"). InnerVolumeSpecName "kube-api-access-m8spw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:17.258311 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.258277    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8spw\" (UniqueName: \"kubernetes.io/projected/36fdc0d0-2fb0-4d96-90e3-09955e545f71-kube-api-access-m8spw\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:01:17.258311 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.258303    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fdc0d0-2fb0-4d96-90e3-09955e545f71-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:01:17.755596 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.755561    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:01:17.775966 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.775933    2559 generic.go:356] "Generic (PLEG): container finished" podID="36fdc0d0-2fb0-4d96-90e3-09955e545f71" containerID="88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679" exitCode=0
Sep 16 13:01:17.776162 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.775979    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"
Sep 16 13:01:17.776162 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.776022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45" event={"ID":"36fdc0d0-2fb0-4d96-90e3-09955e545f71","Type":"ContainerDied","Data":"88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679"}
Sep 16 13:01:17.776162 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.776071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45" event={"ID":"36fdc0d0-2fb0-4d96-90e3-09955e545f71","Type":"ContainerDied","Data":"9e6767da2bb1042132952e6b5a0e0d396cf851acf72464437c010041063a56e1"}
Sep 16 13:01:17.776162 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.776093    2559 scope.go:117] "RemoveContainer" containerID="88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679"
Sep 16 13:01:17.787065 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.787032    2559 scope.go:117] "RemoveContainer" containerID="88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679"
Sep 16 13:01:17.787501 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:01:17.787441    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679\": container with ID starting with 88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679 not found: ID does not exist" containerID="88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679"
Sep 16 13:01:17.787501 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.787479    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679"} err="failed to get container status \"88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679\": rpc error: code = NotFound desc = could not find container \"88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679\": container with ID starting with 88157ffa478da39e7913cb76e42d96628ec141b71209ded7aa3a049776319679 not found: ID does not exist"
Sep 16 13:01:17.800160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.800131    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"]
Sep 16 13:01:17.806868 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:17.806829    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-586bab-89b68857c-jtc45"]
Sep 16 13:01:18.414659 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:18.414623    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36fdc0d0-2fb0-4d96-90e3-09955e545f71" path="/var/lib/kubelet/pods/36fdc0d0-2fb0-4d96-90e3-09955e545f71/volumes"
Sep 16 13:01:35.876360 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:35.876310    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"]
Sep 16 13:01:35.876898 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:35.876583    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2" podUID="a958870d-94d9-428a-a03a-00f95f3fb111" containerName="payload-processing" containerID="cri-o://41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843" gracePeriod=30
Sep 16 13:01:36.138536 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.138468    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 13:01:36.248456 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.248427    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqk88\" (UniqueName: \"kubernetes.io/projected/a958870d-94d9-428a-a03a-00f95f3fb111-kube-api-access-kqk88\") pod \"a958870d-94d9-428a-a03a-00f95f3fb111\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") "
Sep 16 13:01:36.248618 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.248543    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a958870d-94d9-428a-a03a-00f95f3fb111-plugins-config-volume\") pod \"a958870d-94d9-428a-a03a-00f95f3fb111\" (UID: \"a958870d-94d9-428a-a03a-00f95f3fb111\") "
Sep 16 13:01:36.248990 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.248964    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a958870d-94d9-428a-a03a-00f95f3fb111-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a958870d-94d9-428a-a03a-00f95f3fb111" (UID: "a958870d-94d9-428a-a03a-00f95f3fb111"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:01:36.250655 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.250628    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a958870d-94d9-428a-a03a-00f95f3fb111-kube-api-access-kqk88" (OuterVolumeSpecName: "kube-api-access-kqk88") pod "a958870d-94d9-428a-a03a-00f95f3fb111" (UID: "a958870d-94d9-428a-a03a-00f95f3fb111"). InnerVolumeSpecName "kube-api-access-kqk88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:01:36.349800 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.349746    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a958870d-94d9-428a-a03a-00f95f3fb111-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:01:36.349800 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.349797    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqk88\" (UniqueName: \"kubernetes.io/projected/a958870d-94d9-428a-a03a-00f95f3fb111-kube-api-access-kqk88\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:01:36.877383 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.877265    2559 generic.go:356] "Generic (PLEG): container finished" podID="a958870d-94d9-428a-a03a-00f95f3fb111" containerID="41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843" exitCode=0
Sep 16 13:01:36.877383 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.877314    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"
Sep 16 13:01:36.877383 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.877358    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2" event={"ID":"a958870d-94d9-428a-a03a-00f95f3fb111","Type":"ContainerDied","Data":"41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843"}
Sep 16 13:01:36.877915 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.877413    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2" event={"ID":"a958870d-94d9-428a-a03a-00f95f3fb111","Type":"ContainerDied","Data":"d2a0727a4943b4066c6fcfc49bdabcc860db343d463e6cf6527f9987095014ef"}
Sep 16 13:01:36.877915 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.877445    2559 scope.go:117] "RemoveContainer" containerID="41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843"
Sep 16 13:01:36.886663 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.886642    2559 scope.go:117] "RemoveContainer" containerID="41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843"
Sep 16 13:01:36.886967 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:01:36.886939    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843\": container with ID starting with 41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843 not found: ID does not exist" containerID="41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843"
Sep 16 13:01:36.887040 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.886968    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843"} err="failed to get container status \"41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843\": rpc error: code = NotFound desc = could not find container \"41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843\": container with ID starting with 41308ee2076bc5d3b74766de912038135f6014927d4bda7e7ab9c2bc8a98f843 not found: ID does not exist"
Sep 16 13:01:36.898610 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.898585    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"]
Sep 16 13:01:36.906499 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:36.906479    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89d6bf-75678b5cdf-6svp2"]
Sep 16 13:01:38.412754 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:38.412721    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a958870d-94d9-428a-a03a-00f95f3fb111" path="/var/lib/kubelet/pods/a958870d-94d9-428a-a03a-00f95f3fb111/volumes"
Sep 16 13:01:45.942670 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:45.942638    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:01:49.602239 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602197    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"]
Sep 16 13:01:49.602694 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602596    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a958870d-94d9-428a-a03a-00f95f3fb111" containerName="payload-processing"
Sep 16 13:01:49.602694 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602607    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a958870d-94d9-428a-a03a-00f95f3fb111" containerName="payload-processing"
Sep 16 13:01:49.602694 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602636    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36fdc0d0-2fb0-4d96-90e3-09955e545f71" containerName="payload-processing"
Sep 16 13:01:49.602694 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602642    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36fdc0d0-2fb0-4d96-90e3-09955e545f71" containerName="payload-processing"
Sep 16 13:01:49.602857 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602702    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36fdc0d0-2fb0-4d96-90e3-09955e545f71" containerName="payload-processing"
Sep 16 13:01:49.602857 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.602713    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a958870d-94d9-428a-a03a-00f95f3fb111" containerName="payload-processing"
Sep 16 13:01:49.605728 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.605712    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.608818 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.608799    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-524adf28\""
Sep 16 13:01:49.609046 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.609029    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-524adf28-dockercfg-q2f8b\""
Sep 16 13:01:49.620380 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.620355    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"]
Sep 16 13:01:49.688594 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.688553    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48c6\" (UniqueName: \"kubernetes.io/projected/489bf437-fd0c-4a2b-8650-e270f9d48500-kube-api-access-h48c6\") pod \"payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") " pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.688594 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.688601    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489bf437-fd0c-4a2b-8650-e270f9d48500-plugins-config-volume\") pod \"payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") " pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.789850 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.789806    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48c6\" (UniqueName: \"kubernetes.io/projected/489bf437-fd0c-4a2b-8650-e270f9d48500-kube-api-access-h48c6\") pod \"payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") " pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.790024 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.789860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489bf437-fd0c-4a2b-8650-e270f9d48500-plugins-config-volume\") pod \"payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") " pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.790562 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.790541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489bf437-fd0c-4a2b-8650-e270f9d48500-plugins-config-volume\") pod \"payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") " pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.799442 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.799419    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48c6\" (UniqueName: \"kubernetes.io/projected/489bf437-fd0c-4a2b-8650-e270f9d48500-kube-api-access-h48c6\") pod \"payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") " pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:49.915642 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:49.915606    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:50.191266 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:50.191237    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"]
Sep 16 13:01:50.192014 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:01:50.191991    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489bf437_fd0c_4a2b_8650_e270f9d48500.slice/crio-b6e964d874ebc7c810e07832251a82e0ff05ff451a3372d326879651adab2ab0 WatchSource:0}: Error finding container b6e964d874ebc7c810e07832251a82e0ff05ff451a3372d326879651adab2ab0: Status 404 returned error can't find the container with id b6e964d874ebc7c810e07832251a82e0ff05ff451a3372d326879651adab2ab0
Sep 16 13:01:50.952422 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:50.952388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf" event={"ID":"489bf437-fd0c-4a2b-8650-e270f9d48500","Type":"ContainerStarted","Data":"0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416"}
Sep 16 13:01:50.952897 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:50.952449    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf" event={"ID":"489bf437-fd0c-4a2b-8650-e270f9d48500","Type":"ContainerStarted","Data":"b6e964d874ebc7c810e07832251a82e0ff05ff451a3372d326879651adab2ab0"}
Sep 16 13:01:50.952897 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:50.952496    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:01:50.974356 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:01:50.974293    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf" podStartSLOduration=1.974274054 podStartE2EDuration="1.974274054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:01:50.972066844 +0000 UTC m=+1413.055509881" watchObservedRunningTime="2026-09-16 13:01:50.974274054 +0000 UTC m=+1413.057717067"
Sep 16 13:02:01.960461 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:01.960426    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:02:21.576456 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:21.576415    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:02:22.276904 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.276869    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"]
Sep 16 13:02:22.281586 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.281568    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.284565 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.284547    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e344f9-serving-cert\""
Sep 16 13:02:22.300204 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.300170    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"]
Sep 16 13:02:22.402085 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.402042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.402283 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.402108    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3918b-ea6d-4352-8d49-74f1d524d007-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.402283 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.402182    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nkn\" (UniqueName: \"kubernetes.io/projected/1aa3918b-ea6d-4352-8d49-74f1d524d007-kube-api-access-b5nkn\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.402283 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.402227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.503072 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.503031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.503274 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.503154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.503274 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:02:22.503179    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-serving-cert: secret "maas-api-e2e-shared-b-w3-e344f9-serving-cert" not found
Sep 16 13:02:22.503274 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.503209    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3918b-ea6d-4352-8d49-74f1d524d007-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.503274 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:02:22.503252    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls podName:1aa3918b-ea6d-4352-8d49-74f1d524d007 nodeName:}" failed. No retries permitted until 2026-09-16 13:02:23.003229493 +0000 UTC m=+1445.086672491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" (UID: "1aa3918b-ea6d-4352-8d49-74f1d524d007") : secret "maas-api-e2e-shared-b-w3-e344f9-serving-cert" not found
Sep 16 13:02:22.503516 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.503295    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nkn\" (UniqueName: \"kubernetes.io/projected/1aa3918b-ea6d-4352-8d49-74f1d524d007-kube-api-access-b5nkn\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.503945 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.503920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3918b-ea6d-4352-8d49-74f1d524d007-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.505763 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.505742    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:22.512116 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:22.512091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nkn\" (UniqueName: \"kubernetes.io/projected/1aa3918b-ea6d-4352-8d49-74f1d524d007-kube-api-access-b5nkn\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:23.008533 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.008496    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:23.011251 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.011223    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:23.150376 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.150335    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"]
Sep 16 13:02:23.154093 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.154061    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.156758 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.156739    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e344f9\""
Sep 16 13:02:23.157090 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.157066    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e344f9-dockercfg-mdbpm\""
Sep 16 13:02:23.164957 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.164934    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"]
Sep 16 13:02:23.192333 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.192297    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:23.211547 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.211522    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhff\" (UniqueName: \"kubernetes.io/projected/ee86e007-bb68-4d4d-8419-fdd892c34d9c-kube-api-access-cwhff\") pod \"payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.211657 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.211610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee86e007-bb68-4d4d-8419-fdd892c34d9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.312149 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.312060    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhff\" (UniqueName: \"kubernetes.io/projected/ee86e007-bb68-4d4d-8419-fdd892c34d9c-kube-api-access-cwhff\") pod \"payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.312149 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.312146    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee86e007-bb68-4d4d-8419-fdd892c34d9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.312814 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.312794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee86e007-bb68-4d4d-8419-fdd892c34d9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.321862 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.321838    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhff\" (UniqueName: \"kubernetes.io/projected/ee86e007-bb68-4d4d-8419-fdd892c34d9c-kube-api-access-cwhff\") pod \"payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.465617 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.465577    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:23.473369 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.473344    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"]
Sep 16 13:02:23.476691 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:02:23.476660    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa3918b_ea6d_4352_8d49_74f1d524d007.slice/crio-4d4db086fcd68c3e3781b549ad8d70d72cc3375f96ff26a60fe4b859bf88c914 WatchSource:0}: Error finding container 4d4db086fcd68c3e3781b549ad8d70d72cc3375f96ff26a60fe4b859bf88c914: Status 404 returned error can't find the container with id 4d4db086fcd68c3e3781b549ad8d70d72cc3375f96ff26a60fe4b859bf88c914
Sep 16 13:02:23.750466 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:23.750441    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"]
Sep 16 13:02:23.751876 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:02:23.751847    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee86e007_bb68_4d4d_8419_fdd892c34d9c.slice/crio-5406de4744a0d53f11642815eca05ed5fe9f774bb0b8ffac6038f0a962ae6dbc WatchSource:0}: Error finding container 5406de4744a0d53f11642815eca05ed5fe9f774bb0b8ffac6038f0a962ae6dbc: Status 404 returned error can't find the container with id 5406de4744a0d53f11642815eca05ed5fe9f774bb0b8ffac6038f0a962ae6dbc
Sep 16 13:02:24.126571 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.126471    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb" event={"ID":"ee86e007-bb68-4d4d-8419-fdd892c34d9c","Type":"ContainerStarted","Data":"a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726"}
Sep 16 13:02:24.127018 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.126573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb" event={"ID":"ee86e007-bb68-4d4d-8419-fdd892c34d9c","Type":"ContainerStarted","Data":"5406de4744a0d53f11642815eca05ed5fe9f774bb0b8ffac6038f0a962ae6dbc"}
Sep 16 13:02:24.127018 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.126608    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:24.128200 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.128176    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" event={"ID":"1aa3918b-ea6d-4352-8d49-74f1d524d007","Type":"ContainerStarted","Data":"59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e"}
Sep 16 13:02:24.128302 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.128216    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" event={"ID":"1aa3918b-ea6d-4352-8d49-74f1d524d007","Type":"ContainerStarted","Data":"4d4db086fcd68c3e3781b549ad8d70d72cc3375f96ff26a60fe4b859bf88c914"}
Sep 16 13:02:24.128302 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.128236    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:24.149213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.149171    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb" podStartSLOduration=1.1491563710000001 podStartE2EDuration="1.149156371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:24.148460126 +0000 UTC m=+1446.231903160" watchObservedRunningTime="2026-09-16 13:02:24.149156371 +0000 UTC m=+1446.232599386"
Sep 16 13:02:24.170152 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:24.170107    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" podStartSLOduration=1.71426312 podStartE2EDuration="2.170093001s" totalImagesPullingTime="455.829881ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:24.169342494 +0000 UTC m=+1446.252785507" watchObservedRunningTime="2026-09-16 13:02:24.170093001 +0000 UTC m=+1446.253536012"
Sep 16 13:02:27.978186 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:27.978146    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"]
Sep 16 13:02:27.982068 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:27.982047    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:27.986064 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:27.986045    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b159da\"/\"kube-root-ca.crt\""
Sep 16 13:02:27.986064 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:27.986060    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b159da\"/\"noten-model-b159da-kserve-self-signed-certs\""
Sep 16 13:02:27.986869 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:27.986854    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b159da\"/\"default-dockercfg-qsc24\""
Sep 16 13:02:27.986953 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:27.986887    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b159da\"/\"openshift-service-ca.crt\""
Sep 16 13:02:28.003199 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.003175    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"]
Sep 16 13:02:28.065695 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.065656    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnn65\" (UniqueName: \"kubernetes.io/projected/673a0695-6787-4d8b-a40b-1b17745d724b-kube-api-access-pnn65\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.065695 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.065696    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-tmp-dir\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.065899 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.065792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-model-cache\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.065899 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.065856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-home\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.065975 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.065914    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-dshm\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.065975 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.065955    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673a0695-6787-4d8b-a40b-1b17745d724b-tls-certs\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.166924 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.166888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-tmp-dir\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.166969    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-model-cache\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167031    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-home\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-dshm\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673a0695-6787-4d8b-a40b-1b17745d724b-tls-certs\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167359 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167174    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnn65\" (UniqueName: \"kubernetes.io/projected/673a0695-6787-4d8b-a40b-1b17745d724b-kube-api-access-pnn65\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167359 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167345    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-tmp-dir\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167477 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167382    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-model-cache\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.167477 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.167405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-home\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.169404 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.169384    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-dshm\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.169698 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.169682    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673a0695-6787-4d8b-a40b-1b17745d724b-tls-certs\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.181781 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.181753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnn65\" (UniqueName: \"kubernetes.io/projected/673a0695-6787-4d8b-a40b-1b17745d724b-kube-api-access-pnn65\") pod \"noten-model-b159da-kserve-66f5ccd7b7-8vxtr\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") " pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.293825 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.293740    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:28.573088 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:28.573066    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"]
Sep 16 13:02:28.574925 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:02:28.574896    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod673a0695_6787_4d8b_a40b_1b17745d724b.slice/crio-0ba1a41b7050c1d8515b05d127e3c37602076f844e01ebb3f2fdd48c8513f012 WatchSource:0}: Error finding container 0ba1a41b7050c1d8515b05d127e3c37602076f844e01ebb3f2fdd48c8513f012: Status 404 returned error can't find the container with id 0ba1a41b7050c1d8515b05d127e3c37602076f844e01ebb3f2fdd48c8513f012
Sep 16 13:02:29.155777 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:29.155739    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr" event={"ID":"673a0695-6787-4d8b-a40b-1b17745d724b","Type":"ContainerStarted","Data":"617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081"}
Sep 16 13:02:29.156156 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:29.155792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr" event={"ID":"673a0695-6787-4d8b-a40b-1b17745d724b","Type":"ContainerStarted","Data":"0ba1a41b7050c1d8515b05d127e3c37602076f844e01ebb3f2fdd48c8513f012"}
Sep 16 13:02:29.179672 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:29.179628    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr" podStartSLOduration=2.1796135899999998 podStartE2EDuration="2.17961359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:02:29.175807486 +0000 UTC m=+1451.259250500" watchObservedRunningTime="2026-09-16 13:02:29.17961359 +0000 UTC m=+1451.263056603"
Sep 16 13:02:30.139395 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:30.139367    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:02:35.135537 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:35.135503    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:02:38.293924 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:38.293894    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:38.294306 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:38.293943    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:38.306202 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:38.306177    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:39.218930 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:39.218900    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:02:55.191067 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:02:55.191031    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:03:07.120065 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:07.120021    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"]
Sep 16 13:03:22.139960 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.139919    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr" podUID="673a0695-6787-4d8b-a40b-1b17745d724b" containerName="main" containerID="cri-o://617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081" gracePeriod=45
Sep 16 13:03:22.409799 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.409726    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:03:22.443490 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.443453    2559 generic.go:356] "Generic (PLEG): container finished" podID="673a0695-6787-4d8b-a40b-1b17745d724b" containerID="617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081" exitCode=0
Sep 16 13:03:22.443662 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.443512    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"
Sep 16 13:03:22.443662 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.443617    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr" event={"ID":"673a0695-6787-4d8b-a40b-1b17745d724b","Type":"ContainerDied","Data":"617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081"}
Sep 16 13:03:22.443662 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.443653    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr" event={"ID":"673a0695-6787-4d8b-a40b-1b17745d724b","Type":"ContainerDied","Data":"0ba1a41b7050c1d8515b05d127e3c37602076f844e01ebb3f2fdd48c8513f012"}
Sep 16 13:03:22.443820 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.443677    2559 scope.go:117] "RemoveContainer" containerID="617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081"
Sep 16 13:03:22.453211 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.453189    2559 scope.go:117] "RemoveContainer" containerID="617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081"
Sep 16 13:03:22.453527 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:03:22.453505    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081\": container with ID starting with 617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081 not found: ID does not exist" containerID="617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081"
Sep 16 13:03:22.453608 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.453538    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081"} err="failed to get container status \"617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081\": rpc error: code = NotFound desc = could not find container \"617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081\": container with ID starting with 617713a4550e7341f5b04c44e57898926efb24c732290ac5616b51a28a1c6081 not found: ID does not exist"
Sep 16 13:03:22.501430 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501400    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnn65\" (UniqueName: \"kubernetes.io/projected/673a0695-6787-4d8b-a40b-1b17745d724b-kube-api-access-pnn65\") pod \"673a0695-6787-4d8b-a40b-1b17745d724b\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") "
Sep 16 13:03:22.501430 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501431    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673a0695-6787-4d8b-a40b-1b17745d724b-tls-certs\") pod \"673a0695-6787-4d8b-a40b-1b17745d724b\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") "
Sep 16 13:03:22.501693 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501466    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-tmp-dir\") pod \"673a0695-6787-4d8b-a40b-1b17745d724b\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") "
Sep 16 13:03:22.501693 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501565    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-model-cache\") pod \"673a0695-6787-4d8b-a40b-1b17745d724b\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") "
Sep 16 13:03:22.501693 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501602    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-home\") pod \"673a0695-6787-4d8b-a40b-1b17745d724b\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") "
Sep 16 13:03:22.501693 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501680    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-dshm\") pod \"673a0695-6787-4d8b-a40b-1b17745d724b\" (UID: \"673a0695-6787-4d8b-a40b-1b17745d724b\") "
Sep 16 13:03:22.501902 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501810    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "673a0695-6787-4d8b-a40b-1b17745d724b" (UID: "673a0695-6787-4d8b-a40b-1b17745d724b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:22.501902 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.501887    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-home" (OuterVolumeSpecName: "home") pod "673a0695-6787-4d8b-a40b-1b17745d724b" (UID: "673a0695-6787-4d8b-a40b-1b17745d724b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:22.502290 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.502267    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:03:22.502415 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.502291    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:03:22.502467 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.502448    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-model-cache" (OuterVolumeSpecName: "model-cache") pod "673a0695-6787-4d8b-a40b-1b17745d724b" (UID: "673a0695-6787-4d8b-a40b-1b17745d724b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:22.503763 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.503743    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673a0695-6787-4d8b-a40b-1b17745d724b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "673a0695-6787-4d8b-a40b-1b17745d724b" (UID: "673a0695-6787-4d8b-a40b-1b17745d724b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:03:22.503866 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.503774    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673a0695-6787-4d8b-a40b-1b17745d724b-kube-api-access-pnn65" (OuterVolumeSpecName: "kube-api-access-pnn65") pod "673a0695-6787-4d8b-a40b-1b17745d724b" (UID: "673a0695-6787-4d8b-a40b-1b17745d724b"). InnerVolumeSpecName "kube-api-access-pnn65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:03:22.503866 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.503823    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-dshm" (OuterVolumeSpecName: "dshm") pod "673a0695-6787-4d8b-a40b-1b17745d724b" (UID: "673a0695-6787-4d8b-a40b-1b17745d724b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:03:22.603588 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.603556    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:03:22.603588 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.603584    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673a0695-6787-4d8b-a40b-1b17745d724b-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:03:22.603588 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.603593    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnn65\" (UniqueName: \"kubernetes.io/projected/673a0695-6787-4d8b-a40b-1b17745d724b-kube-api-access-pnn65\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:03:22.603822 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.603602    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673a0695-6787-4d8b-a40b-1b17745d724b-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:03:22.766483 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.766440    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"]
Sep 16 13:03:22.768807 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:22.768786    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-b159da/noten-model-b159da-kserve-66f5ccd7b7-8vxtr"]
Sep 16 13:03:24.412822 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:24.412789    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673a0695-6787-4d8b-a40b-1b17745d724b" path="/var/lib/kubelet/pods/673a0695-6787-4d8b-a40b-1b17745d724b/volumes"
Sep 16 13:03:35.578558 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.578525    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"]
Sep 16 13:03:35.579025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.578929    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673a0695-6787-4d8b-a40b-1b17745d724b" containerName="main"
Sep 16 13:03:35.579025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.578941    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="673a0695-6787-4d8b-a40b-1b17745d724b" containerName="main"
Sep 16 13:03:35.579025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.579015    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="673a0695-6787-4d8b-a40b-1b17745d724b" containerName="main"
Sep 16 13:03:35.583312 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.583294    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.586583 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.586564    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-524adf28\"/\"openshift-service-ca.crt\""
Sep 16 13:03:35.586711 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.586601    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-524adf28\"/\"kube-root-ca.crt\""
Sep 16 13:03:35.586711 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.586569    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-524adf28\"/\"default-dockercfg-ss9zp\""
Sep 16 13:03:35.586711 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.586569    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-524adf28\"/\"e2e-lifecycle-model-524adf28-kserve-self-signed-certs\""
Sep 16 13:03:35.591370 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.591348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"]
Sep 16 13:03:35.736529 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.736496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d00795-2b43-49b0-9652-099e78bc29bd-tls-certs\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.736529 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.736531    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-home\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.736779 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.736557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x54l\" (UniqueName: \"kubernetes.io/projected/f5d00795-2b43-49b0-9652-099e78bc29bd-kube-api-access-5x54l\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.736779 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.736619    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-model-cache\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.736779 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.736684    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-tmp-dir\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.736779 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.736736    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-dshm\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838133    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d00795-2b43-49b0-9652-099e78bc29bd-tls-certs\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838178    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-home\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838205    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x54l\" (UniqueName: \"kubernetes.io/projected/f5d00795-2b43-49b0-9652-099e78bc29bd-kube-api-access-5x54l\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838480 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838226    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-model-cache\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838480 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838253    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-tmp-dir\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838480 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838300    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-dshm\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838655 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838627    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-home\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838748 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-tmp-dir\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.838817 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.838746    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-model-cache\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.840757 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.840735    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-dshm\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.841121 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.841099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d00795-2b43-49b0-9652-099e78bc29bd-tls-certs\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.851544 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.851518    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x54l\" (UniqueName: \"kubernetes.io/projected/f5d00795-2b43-49b0-9652-099e78bc29bd-kube-api-access-5x54l\") pod \"e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") " pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:35.892553 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:35.892526    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:36.188172 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:03:36.188139    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5d00795_2b43_49b0_9652_099e78bc29bd.slice/crio-2b9235357220341278d6698a51b8924a579a32d21780ad612dae6cbb80417830 WatchSource:0}: Error finding container 2b9235357220341278d6698a51b8924a579a32d21780ad612dae6cbb80417830: Status 404 returned error can't find the container with id 2b9235357220341278d6698a51b8924a579a32d21780ad612dae6cbb80417830
Sep 16 13:03:36.196886 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:36.196864    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"]
Sep 16 13:03:36.517169 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:36.517068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt" event={"ID":"f5d00795-2b43-49b0-9652-099e78bc29bd","Type":"ContainerStarted","Data":"8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1"}
Sep 16 13:03:36.517169 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:36.517111    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt" event={"ID":"f5d00795-2b43-49b0-9652-099e78bc29bd","Type":"ContainerStarted","Data":"2b9235357220341278d6698a51b8924a579a32d21780ad612dae6cbb80417830"}
Sep 16 13:03:36.539768 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:36.539723    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt" podStartSLOduration=1.539708642 podStartE2EDuration="1.539708642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:03:36.538662431 +0000 UTC m=+1518.622105458" watchObservedRunningTime="2026-09-16 13:03:36.539708642 +0000 UTC m=+1518.623151656"
Sep 16 13:03:45.892710 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:45.892623    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:45.892710 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:45.892673    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:45.905160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:45.905138    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:46.580853 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:46.580824    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:03:47.970614 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:03:47.970583    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:04:17.369480 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:17.369437    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"]
Sep 16 13:04:17.382457 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:17.382430    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"]
Sep 16 13:04:24.057366 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:24.057313    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:04:28.368583 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.368546    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"]
Sep 16 13:04:28.377344 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.377303    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-490797-29826060-64m47"]
Sep 16 13:04:28.414786 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.414749    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1be646d-4422-42e8-91b3-0b9aa3f4685b" path="/var/lib/kubelet/pods/f1be646d-4422-42e8-91b3-0b9aa3f4685b/volumes"
Sep 16 13:04:28.415087 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.415070    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"]
Sep 16 13:04:28.415371 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.415347    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b" podUID="fd5eba89-f4e7-41eb-810b-d7936c96740b" containerName="payload-processing" containerID="cri-o://8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914" gracePeriod=30
Sep 16 13:04:28.661282 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.661255    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 13:04:28.770040 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.770015    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2rfc\" (UniqueName: \"kubernetes.io/projected/fd5eba89-f4e7-41eb-810b-d7936c96740b-kube-api-access-d2rfc\") pod \"fd5eba89-f4e7-41eb-810b-d7936c96740b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") "
Sep 16 13:04:28.770224 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.770098    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5eba89-f4e7-41eb-810b-d7936c96740b-plugins-config-volume\") pod \"fd5eba89-f4e7-41eb-810b-d7936c96740b\" (UID: \"fd5eba89-f4e7-41eb-810b-d7936c96740b\") "
Sep 16 13:04:28.770514 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.770488    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5eba89-f4e7-41eb-810b-d7936c96740b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd5eba89-f4e7-41eb-810b-d7936c96740b" (UID: "fd5eba89-f4e7-41eb-810b-d7936c96740b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:04:28.772172 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.772153    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5eba89-f4e7-41eb-810b-d7936c96740b-kube-api-access-d2rfc" (OuterVolumeSpecName: "kube-api-access-d2rfc") pod "fd5eba89-f4e7-41eb-810b-d7936c96740b" (UID: "fd5eba89-f4e7-41eb-810b-d7936c96740b"). InnerVolumeSpecName "kube-api-access-d2rfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:28.797487 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.797456    2559 generic.go:356] "Generic (PLEG): container finished" podID="fd5eba89-f4e7-41eb-810b-d7936c96740b" containerID="8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914" exitCode=0
Sep 16 13:04:28.797620 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.797488    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b" event={"ID":"fd5eba89-f4e7-41eb-810b-d7936c96740b","Type":"ContainerDied","Data":"8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914"}
Sep 16 13:04:28.797620 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.797505    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"
Sep 16 13:04:28.797620 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.797537    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b" event={"ID":"fd5eba89-f4e7-41eb-810b-d7936c96740b","Type":"ContainerDied","Data":"442f7167b693e46b991dac553c17885a425581cc043f4e77e4bf0ca242ad0543"}
Sep 16 13:04:28.797620 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.797567    2559 scope.go:117] "RemoveContainer" containerID="8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914"
Sep 16 13:04:28.808267 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.808251    2559 scope.go:117] "RemoveContainer" containerID="8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914"
Sep 16 13:04:28.808553 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:04:28.808533    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914\": container with ID starting with 8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914 not found: ID does not exist" containerID="8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914"
Sep 16 13:04:28.808637 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.808566    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914"} err="failed to get container status \"8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914\": rpc error: code = NotFound desc = could not find container \"8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914\": container with ID starting with 8dcf6426674fc65ccc13339ae7a85b348ba6442f218d25e58ecc71024a188914 not found: ID does not exist"
Sep 16 13:04:28.823285 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.823262    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"]
Sep 16 13:04:28.827314 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.827294    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-490797-7b949f5899-6pm4b"]
Sep 16 13:04:28.871423 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.871392    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2rfc\" (UniqueName: \"kubernetes.io/projected/fd5eba89-f4e7-41eb-810b-d7936c96740b-kube-api-access-d2rfc\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:28.871423 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:28.871418    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5eba89-f4e7-41eb-810b-d7936c96740b-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:30.412936 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:30.412886    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5eba89-f4e7-41eb-810b-d7936c96740b" path="/var/lib/kubelet/pods/fd5eba89-f4e7-41eb-810b-d7936c96740b/volumes"
Sep 16 13:04:32.389554 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.389496    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z" podUID="b253a0c6-3665-4d05-9283-13ec07bfa15b" containerName="main" containerID="cri-o://30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102" gracePeriod=45
Sep 16 13:04:32.401751 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.401719    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9" podUID="092aa52a-34a3-448d-8a29-ba6d7730c70b" containerName="main" containerID="cri-o://0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab" gracePeriod=45
Sep 16 13:04:32.665742 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.665719    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:04:32.668532 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.668513    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 13:04:32.710012 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.709987    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-model-cache\") pod \"092aa52a-34a3-448d-8a29-ba6d7730c70b\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") "
Sep 16 13:04:32.710189 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710066    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-home\") pod \"b253a0c6-3665-4d05-9283-13ec07bfa15b\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") "
Sep 16 13:04:32.710189 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710089    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m8bn\" (UniqueName: \"kubernetes.io/projected/092aa52a-34a3-448d-8a29-ba6d7730c70b-kube-api-access-8m8bn\") pod \"092aa52a-34a3-448d-8a29-ba6d7730c70b\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") "
Sep 16 13:04:32.710189 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710128    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b253a0c6-3665-4d05-9283-13ec07bfa15b-tls-certs\") pod \"b253a0c6-3665-4d05-9283-13ec07bfa15b\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") "
Sep 16 13:04:32.710189 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710152    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-tmp-dir\") pod \"092aa52a-34a3-448d-8a29-ba6d7730c70b\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") "
Sep 16 13:04:32.710189 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710186    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-tmp-dir\") pod \"b253a0c6-3665-4d05-9283-13ec07bfa15b\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710218    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4jrj\" (UniqueName: \"kubernetes.io/projected/b253a0c6-3665-4d05-9283-13ec07bfa15b-kube-api-access-v4jrj\") pod \"b253a0c6-3665-4d05-9283-13ec07bfa15b\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710264    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-model-cache\") pod \"b253a0c6-3665-4d05-9283-13ec07bfa15b\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710293    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-model-cache" (OuterVolumeSpecName: "model-cache") pod "092aa52a-34a3-448d-8a29-ba6d7730c70b" (UID: "092aa52a-34a3-448d-8a29-ba6d7730c70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710333    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-home" (OuterVolumeSpecName: "home") pod "b253a0c6-3665-4d05-9283-13ec07bfa15b" (UID: "b253a0c6-3665-4d05-9283-13ec07bfa15b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-dshm\") pod \"092aa52a-34a3-448d-8a29-ba6d7730c70b\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710414    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-home\") pod \"092aa52a-34a3-448d-8a29-ba6d7730c70b\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-dshm\") pod \"b253a0c6-3665-4d05-9283-13ec07bfa15b\" (UID: \"b253a0c6-3665-4d05-9283-13ec07bfa15b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710485    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092aa52a-34a3-448d-8a29-ba6d7730c70b-tls-certs\") pod \"092aa52a-34a3-448d-8a29-ba6d7730c70b\" (UID: \"092aa52a-34a3-448d-8a29-ba6d7730c70b\") "
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710553    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "092aa52a-34a3-448d-8a29-ba6d7730c70b" (UID: "092aa52a-34a3-448d-8a29-ba6d7730c70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710562    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-model-cache" (OuterVolumeSpecName: "model-cache") pod "b253a0c6-3665-4d05-9283-13ec07bfa15b" (UID: "b253a0c6-3665-4d05-9283-13ec07bfa15b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710713    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-home" (OuterVolumeSpecName: "home") pod "092aa52a-34a3-448d-8a29-ba6d7730c70b" (UID: "092aa52a-34a3-448d-8a29-ba6d7730c70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.710882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710801    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b253a0c6-3665-4d05-9283-13ec07bfa15b" (UID: "b253a0c6-3665-4d05-9283-13ec07bfa15b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.711811 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710923    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.711811 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710944    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.711811 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710959    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.711811 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710970    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.711811 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710982    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.711811 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.710994    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.713917 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.713888    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b253a0c6-3665-4d05-9283-13ec07bfa15b-kube-api-access-v4jrj" (OuterVolumeSpecName: "kube-api-access-v4jrj") pod "b253a0c6-3665-4d05-9283-13ec07bfa15b" (UID: "b253a0c6-3665-4d05-9283-13ec07bfa15b"). InnerVolumeSpecName "kube-api-access-v4jrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:32.714035 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.713963    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092aa52a-34a3-448d-8a29-ba6d7730c70b-kube-api-access-8m8bn" (OuterVolumeSpecName: "kube-api-access-8m8bn") pod "092aa52a-34a3-448d-8a29-ba6d7730c70b" (UID: "092aa52a-34a3-448d-8a29-ba6d7730c70b"). InnerVolumeSpecName "kube-api-access-8m8bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:04:32.714035 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.714000    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-dshm" (OuterVolumeSpecName: "dshm") pod "092aa52a-34a3-448d-8a29-ba6d7730c70b" (UID: "092aa52a-34a3-448d-8a29-ba6d7730c70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.714167 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.714101    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-dshm" (OuterVolumeSpecName: "dshm") pod "b253a0c6-3665-4d05-9283-13ec07bfa15b" (UID: "b253a0c6-3665-4d05-9283-13ec07bfa15b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:04:32.714355 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.714314    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/092aa52a-34a3-448d-8a29-ba6d7730c70b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "092aa52a-34a3-448d-8a29-ba6d7730c70b" (UID: "092aa52a-34a3-448d-8a29-ba6d7730c70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:32.714438 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.714357    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b253a0c6-3665-4d05-9283-13ec07bfa15b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b253a0c6-3665-4d05-9283-13ec07bfa15b" (UID: "b253a0c6-3665-4d05-9283-13ec07bfa15b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:04:32.812466 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.812435    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/092aa52a-34a3-448d-8a29-ba6d7730c70b-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.812466 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.812468    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b253a0c6-3665-4d05-9283-13ec07bfa15b-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.812653 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.812481    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/092aa52a-34a3-448d-8a29-ba6d7730c70b-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.812653 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.812497    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m8bn\" (UniqueName: \"kubernetes.io/projected/092aa52a-34a3-448d-8a29-ba6d7730c70b-kube-api-access-8m8bn\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.812653 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.812512    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b253a0c6-3665-4d05-9283-13ec07bfa15b-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.812653 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.812528    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4jrj\" (UniqueName: \"kubernetes.io/projected/b253a0c6-3665-4d05-9283-13ec07bfa15b-kube-api-access-v4jrj\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:04:32.821911 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.821884    2559 generic.go:356] "Generic (PLEG): container finished" podID="b253a0c6-3665-4d05-9283-13ec07bfa15b" containerID="30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102" exitCode=0
Sep 16 13:04:32.822075 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.821942    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"
Sep 16 13:04:32.822075 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.821971    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z" event={"ID":"b253a0c6-3665-4d05-9283-13ec07bfa15b","Type":"ContainerDied","Data":"30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102"}
Sep 16 13:04:32.822075 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.822011    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z" event={"ID":"b253a0c6-3665-4d05-9283-13ec07bfa15b","Type":"ContainerDied","Data":"37d4e9706334d3a146851a74489c210561daee360d2585b934de891db8c3aecf"}
Sep 16 13:04:32.822075 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.822036    2559 scope.go:117] "RemoveContainer" containerID="30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102"
Sep 16 13:04:32.823532 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.823481    2559 generic.go:356] "Generic (PLEG): container finished" podID="092aa52a-34a3-448d-8a29-ba6d7730c70b" containerID="0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab" exitCode=0
Sep 16 13:04:32.823633 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.823540    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"
Sep 16 13:04:32.823633 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.823541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9" event={"ID":"092aa52a-34a3-448d-8a29-ba6d7730c70b","Type":"ContainerDied","Data":"0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab"}
Sep 16 13:04:32.823633 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.823577    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9" event={"ID":"092aa52a-34a3-448d-8a29-ba6d7730c70b","Type":"ContainerDied","Data":"92e6ddf8f6006e2062189f10357a9d1c5e71f451f0b40c54d1c067fc3a24d254"}
Sep 16 13:04:32.833153 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.833134    2559 scope.go:117] "RemoveContainer" containerID="30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102"
Sep 16 13:04:32.833503 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:04:32.833481    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102\": container with ID starting with 30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102 not found: ID does not exist" containerID="30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102"
Sep 16 13:04:32.833604 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.833511    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102"} err="failed to get container status \"30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102\": rpc error: code = NotFound desc = could not find container \"30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102\": container with ID starting with 30e339db0dda035b0638b4d07301e96548cef4038e662cfa9816e50e65f87102 not found: ID does not exist"
Sep 16 13:04:32.833604 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.833529    2559 scope.go:117] "RemoveContainer" containerID="0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab"
Sep 16 13:04:32.841985 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.841970    2559 scope.go:117] "RemoveContainer" containerID="0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab"
Sep 16 13:04:32.842248 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:04:32.842230    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab\": container with ID starting with 0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab not found: ID does not exist" containerID="0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab"
Sep 16 13:04:32.842296 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.842256    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab"} err="failed to get container status \"0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab\": rpc error: code = NotFound desc = could not find container \"0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab\": container with ID starting with 0d11a05d78b1dd2ce442ea0a8a96581d1537b5b09115c2a7c85221f86bc153ab not found: ID does not exist"
Sep 16 13:04:32.853236 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.853212    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"]
Sep 16 13:04:32.855270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.855252    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-490797/premium-w0-490797-kserve-86cb4454b-n52g9"]
Sep 16 13:04:32.879460 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.879433    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"]
Sep 16 13:04:32.881547 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:32.881530    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-490797/distinct-w0-490797-kserve-84869b6957-lqf8z"]
Sep 16 13:04:34.412536 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:34.412505    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092aa52a-34a3-448d-8a29-ba6d7730c70b" path="/var/lib/kubelet/pods/092aa52a-34a3-448d-8a29-ba6d7730c70b/volumes"
Sep 16 13:04:34.412913 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:04:34.412900    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b253a0c6-3665-4d05-9283-13ec07bfa15b" path="/var/lib/kubelet/pods/b253a0c6-3665-4d05-9283-13ec07bfa15b/volumes"
Sep 16 13:05:01.855528 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:01.855491    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:05:12.160244 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:12.160170    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"]
Sep 16 13:05:23.575516 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.575474    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"]
Sep 16 13:05:23.576035 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.575760    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf" podUID="489bf437-fd0c-4a2b-8650-e270f9d48500" containerName="payload-processing" containerID="cri-o://0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416" gracePeriod=30
Sep 16 13:05:23.829566 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.829505    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:05:23.918232 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.918201    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489bf437-fd0c-4a2b-8650-e270f9d48500-plugins-config-volume\") pod \"489bf437-fd0c-4a2b-8650-e270f9d48500\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") "
Sep 16 13:05:23.918511 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.918268    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h48c6\" (UniqueName: \"kubernetes.io/projected/489bf437-fd0c-4a2b-8650-e270f9d48500-kube-api-access-h48c6\") pod \"489bf437-fd0c-4a2b-8650-e270f9d48500\" (UID: \"489bf437-fd0c-4a2b-8650-e270f9d48500\") "
Sep 16 13:05:23.918655 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.918632    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489bf437-fd0c-4a2b-8650-e270f9d48500-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "489bf437-fd0c-4a2b-8650-e270f9d48500" (UID: "489bf437-fd0c-4a2b-8650-e270f9d48500"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:05:23.920393 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:23.920372    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489bf437-fd0c-4a2b-8650-e270f9d48500-kube-api-access-h48c6" (OuterVolumeSpecName: "kube-api-access-h48c6") pod "489bf437-fd0c-4a2b-8650-e270f9d48500" (UID: "489bf437-fd0c-4a2b-8650-e270f9d48500"). InnerVolumeSpecName "kube-api-access-h48c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:24.019920 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.019891    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48c6\" (UniqueName: \"kubernetes.io/projected/489bf437-fd0c-4a2b-8650-e270f9d48500-kube-api-access-h48c6\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:24.019920 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.019919    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489bf437-fd0c-4a2b-8650-e270f9d48500-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:24.089385 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.089290    2559 generic.go:356] "Generic (PLEG): container finished" podID="489bf437-fd0c-4a2b-8650-e270f9d48500" containerID="0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416" exitCode=0
Sep 16 13:05:24.089385 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.089365    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"
Sep 16 13:05:24.089553 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.089387    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf" event={"ID":"489bf437-fd0c-4a2b-8650-e270f9d48500","Type":"ContainerDied","Data":"0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416"}
Sep 16 13:05:24.089553 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.089448    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf" event={"ID":"489bf437-fd0c-4a2b-8650-e270f9d48500","Type":"ContainerDied","Data":"b6e964d874ebc7c810e07832251a82e0ff05ff451a3372d326879651adab2ab0"}
Sep 16 13:05:24.089553 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.089478    2559 scope.go:117] "RemoveContainer" containerID="0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416"
Sep 16 13:05:24.098952 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.098936    2559 scope.go:117] "RemoveContainer" containerID="0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416"
Sep 16 13:05:24.099215 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:05:24.099197    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416\": container with ID starting with 0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416 not found: ID does not exist" containerID="0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416"
Sep 16 13:05:24.099262 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.099225    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416"} err="failed to get container status \"0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416\": rpc error: code = NotFound desc = could not find container \"0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416\": container with ID starting with 0151b89c35d17fa7ed436c08a4439ea6599c003e842e61d6c95e6f4ece0df416 not found: ID does not exist"
Sep 16 13:05:24.110040 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.110021    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"]
Sep 16 13:05:24.114613 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.114593    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-524adf28-57ff794f57-hk4zf"]
Sep 16 13:05:24.411829 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:24.411744    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489bf437-fd0c-4a2b-8650-e270f9d48500" path="/var/lib/kubelet/pods/489bf437-fd0c-4a2b-8650-e270f9d48500/volumes"
Sep 16 13:05:25.755740 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:25.755705    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:05:27.179361 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.179304    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2" podUID="7f7f359c-0d64-4544-a92c-b7918f2f8bd0" containerName="main" containerID="cri-o://83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9" gracePeriod=45
Sep 16 13:05:27.430314 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.430257    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:05:27.554853 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.554823    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tls-certs\") pod \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") "
Sep 16 13:05:27.555004 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.554878    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tmp-dir\") pod \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") "
Sep 16 13:05:27.555004 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.554941    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rncgz\" (UniqueName: \"kubernetes.io/projected/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-kube-api-access-rncgz\") pod \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") "
Sep 16 13:05:27.555004 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.554962    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-model-cache\") pod \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") "
Sep 16 13:05:27.555124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-dshm\") pod \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") "
Sep 16 13:05:27.555124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555031    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-home\") pod \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\" (UID: \"7f7f359c-0d64-4544-a92c-b7918f2f8bd0\") "
Sep 16 13:05:27.555241 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555184    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f7f359c-0d64-4544-a92c-b7918f2f8bd0" (UID: "7f7f359c-0d64-4544-a92c-b7918f2f8bd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:27.555299 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555266    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f7f359c-0d64-4544-a92c-b7918f2f8bd0" (UID: "7f7f359c-0d64-4544-a92c-b7918f2f8bd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:27.555396 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555371    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-home" (OuterVolumeSpecName: "home") pod "7f7f359c-0d64-4544-a92c-b7918f2f8bd0" (UID: "7f7f359c-0d64-4544-a92c-b7918f2f8bd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:27.555458 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555438    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.555497 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.555457    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.557375 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.557356    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-dshm" (OuterVolumeSpecName: "dshm") pod "7f7f359c-0d64-4544-a92c-b7918f2f8bd0" (UID: "7f7f359c-0d64-4544-a92c-b7918f2f8bd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:27.557571 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.557552    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f7f359c-0d64-4544-a92c-b7918f2f8bd0" (UID: "7f7f359c-0d64-4544-a92c-b7918f2f8bd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:27.557675 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.557587    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-kube-api-access-rncgz" (OuterVolumeSpecName: "kube-api-access-rncgz") pod "7f7f359c-0d64-4544-a92c-b7918f2f8bd0" (UID: "7f7f359c-0d64-4544-a92c-b7918f2f8bd0"). InnerVolumeSpecName "kube-api-access-rncgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:27.656630 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.656595    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.656630 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.656618    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.656630 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.656629    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncgz\" (UniqueName: \"kubernetes.io/projected/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-kube-api-access-rncgz\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:27.656630 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:27.656638    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f7f359c-0d64-4544-a92c-b7918f2f8bd0-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:28.113553 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.113516    2559 generic.go:356] "Generic (PLEG): container finished" podID="7f7f359c-0d64-4544-a92c-b7918f2f8bd0" containerID="83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9" exitCode=0
Sep 16 13:05:28.113743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.113563    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"
Sep 16 13:05:28.113743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.113612    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2" event={"ID":"7f7f359c-0d64-4544-a92c-b7918f2f8bd0","Type":"ContainerDied","Data":"83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9"}
Sep 16 13:05:28.113743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.113650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2" event={"ID":"7f7f359c-0d64-4544-a92c-b7918f2f8bd0","Type":"ContainerDied","Data":"2b07fe3bf389c9dda40d5b89c52cbd22a2d3acf24e99436793f1d8ee50ed84b1"}
Sep 16 13:05:28.113743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.113675    2559 scope.go:117] "RemoveContainer" containerID="83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9"
Sep 16 13:05:28.125563 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.125536    2559 scope.go:117] "RemoveContainer" containerID="83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9"
Sep 16 13:05:28.125820 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:05:28.125803    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9\": container with ID starting with 83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9 not found: ID does not exist" containerID="83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9"
Sep 16 13:05:28.125864 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.125828    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9"} err="failed to get container status \"83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9\": rpc error: code = NotFound desc = could not find container \"83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9\": container with ID starting with 83e30d0aac0bccde15efeebfd0d1ceabfcc9e45029e0363bf3e54dfb1c3c95a9 not found: ID does not exist"
Sep 16 13:05:28.134226 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.134202    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"]
Sep 16 13:05:28.138446 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.138427    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/e2e-conflict-a-32338a53-kserve-66bbc54b9-5lql2"]
Sep 16 13:05:28.413381 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:28.413359    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f7f359c-0d64-4544-a92c-b7918f2f8bd0" path="/var/lib/kubelet/pods/7f7f359c-0d64-4544-a92c-b7918f2f8bd0/volumes"
Sep 16 13:05:31.935759 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:31.935727    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"]
Sep 16 13:05:46.955330 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:46.955266    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt" podUID="f5d00795-2b43-49b0-9652-099e78bc29bd" containerName="main" containerID="cri-o://8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1" gracePeriod=45
Sep 16 13:05:47.205468 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.205412    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:05:47.212887 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.212855    2559 generic.go:356] "Generic (PLEG): container finished" podID="f5d00795-2b43-49b0-9652-099e78bc29bd" containerID="8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1" exitCode=0
Sep 16 13:05:47.212887 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.212886    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"
Sep 16 13:05:47.213033 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.212890    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt" event={"ID":"f5d00795-2b43-49b0-9652-099e78bc29bd","Type":"ContainerDied","Data":"8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1"}
Sep 16 13:05:47.213033 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.212978    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt" event={"ID":"f5d00795-2b43-49b0-9652-099e78bc29bd","Type":"ContainerDied","Data":"2b9235357220341278d6698a51b8924a579a32d21780ad612dae6cbb80417830"}
Sep 16 13:05:47.213033 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.212993    2559 scope.go:117] "RemoveContainer" containerID="8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1"
Sep 16 13:05:47.221684 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.221667    2559 scope.go:117] "RemoveContainer" containerID="8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1"
Sep 16 13:05:47.221965 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:05:47.221945    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1\": container with ID starting with 8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1 not found: ID does not exist" containerID="8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1"
Sep 16 13:05:47.222023 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.221974    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1"} err="failed to get container status \"8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1\": rpc error: code = NotFound desc = could not find container \"8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1\": container with ID starting with 8df0c5d9759f7444212b16bb5554ab85e48e26238da6c50b681d5a21c68401f1 not found: ID does not exist"
Sep 16 13:05:47.235202 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235181    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-dshm\") pod \"f5d00795-2b43-49b0-9652-099e78bc29bd\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") "
Sep 16 13:05:47.235374 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235232    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-home\") pod \"f5d00795-2b43-49b0-9652-099e78bc29bd\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") "
Sep 16 13:05:47.235374 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235266    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-tmp-dir\") pod \"f5d00795-2b43-49b0-9652-099e78bc29bd\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") "
Sep 16 13:05:47.235497 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235376    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-model-cache\") pod \"f5d00795-2b43-49b0-9652-099e78bc29bd\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") "
Sep 16 13:05:47.235497 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235432    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d00795-2b43-49b0-9652-099e78bc29bd-tls-certs\") pod \"f5d00795-2b43-49b0-9652-099e78bc29bd\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") "
Sep 16 13:05:47.235497 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235464    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x54l\" (UniqueName: \"kubernetes.io/projected/f5d00795-2b43-49b0-9652-099e78bc29bd-kube-api-access-5x54l\") pod \"f5d00795-2b43-49b0-9652-099e78bc29bd\" (UID: \"f5d00795-2b43-49b0-9652-099e78bc29bd\") "
Sep 16 13:05:47.235661 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235541    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-home" (OuterVolumeSpecName: "home") pod "f5d00795-2b43-49b0-9652-099e78bc29bd" (UID: "f5d00795-2b43-49b0-9652-099e78bc29bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:47.235661 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235571    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5d00795-2b43-49b0-9652-099e78bc29bd" (UID: "f5d00795-2b43-49b0-9652-099e78bc29bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:47.235661 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235649    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5d00795-2b43-49b0-9652-099e78bc29bd" (UID: "f5d00795-2b43-49b0-9652-099e78bc29bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:47.235833 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235789    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:47.235833 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235805    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:47.235833 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.235814    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:47.237706 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.237684    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d00795-2b43-49b0-9652-099e78bc29bd-kube-api-access-5x54l" (OuterVolumeSpecName: "kube-api-access-5x54l") pod "f5d00795-2b43-49b0-9652-099e78bc29bd" (UID: "f5d00795-2b43-49b0-9652-099e78bc29bd"). InnerVolumeSpecName "kube-api-access-5x54l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:05:47.238163 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.238143    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-dshm" (OuterVolumeSpecName: "dshm") pod "f5d00795-2b43-49b0-9652-099e78bc29bd" (UID: "f5d00795-2b43-49b0-9652-099e78bc29bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:05:47.238253 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.238168    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d00795-2b43-49b0-9652-099e78bc29bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5d00795-2b43-49b0-9652-099e78bc29bd" (UID: "f5d00795-2b43-49b0-9652-099e78bc29bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:05:47.336888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.336854    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d00795-2b43-49b0-9652-099e78bc29bd-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:47.336888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.336883    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x54l\" (UniqueName: \"kubernetes.io/projected/f5d00795-2b43-49b0-9652-099e78bc29bd-kube-api-access-5x54l\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:47.336888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.336894    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d00795-2b43-49b0-9652-099e78bc29bd-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:05:47.534955 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.534925    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"]
Sep 16 13:05:47.538927 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.538904    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-524adf28/e2e-lifecycle-model-524adf28-kserve-8669999b59-7fwqt"]
Sep 16 13:05:47.856652 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:47.856572    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:05:48.411741 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:48.411707    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d00795-2b43-49b0-9652-099e78bc29bd" path="/var/lib/kubelet/pods/f5d00795-2b43-49b0-9652-099e78bc29bd/volumes"
Sep 16 13:05:50.451633 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.451605    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"]
Sep 16 13:05:50.452005 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.451986    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489bf437-fd0c-4a2b-8650-e270f9d48500" containerName="payload-processing"
Sep 16 13:05:50.452005 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.451999    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="489bf437-fd0c-4a2b-8650-e270f9d48500" containerName="payload-processing"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452014    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f7f359c-0d64-4544-a92c-b7918f2f8bd0" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452021    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f7f359c-0d64-4544-a92c-b7918f2f8bd0" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452027    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092aa52a-34a3-448d-8a29-ba6d7730c70b" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452033    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="092aa52a-34a3-448d-8a29-ba6d7730c70b" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452039    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b253a0c6-3665-4d05-9283-13ec07bfa15b" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452044    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b253a0c6-3665-4d05-9283-13ec07bfa15b" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452063    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d00795-2b43-49b0-9652-099e78bc29bd" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452068    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d00795-2b43-49b0-9652-099e78bc29bd" containerName="main"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452078    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5eba89-f4e7-41eb-810b-d7936c96740b" containerName="payload-processing"
Sep 16 13:05:50.452095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452083    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5eba89-f4e7-41eb-810b-d7936c96740b" containerName="payload-processing"
Sep 16 13:05:50.452453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452136    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b253a0c6-3665-4d05-9283-13ec07bfa15b" containerName="main"
Sep 16 13:05:50.452453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452143    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5eba89-f4e7-41eb-810b-d7936c96740b" containerName="payload-processing"
Sep 16 13:05:50.452453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452148    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="092aa52a-34a3-448d-8a29-ba6d7730c70b" containerName="main"
Sep 16 13:05:50.452453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452153    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d00795-2b43-49b0-9652-099e78bc29bd" containerName="main"
Sep 16 13:05:50.452453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452158    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f7f359c-0d64-4544-a92c-b7918f2f8bd0" containerName="main"
Sep 16 13:05:50.452453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.452165    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="489bf437-fd0c-4a2b-8650-e270f9d48500" containerName="payload-processing"
Sep 16 13:05:50.455220 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.455205    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.457550 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.457528    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"openshift-service-ca.crt\""
Sep 16 13:05:50.458278 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.458256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"default-dockercfg-gwp2c\""
Sep 16 13:05:50.458404 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.458293    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"auth-test-model-e344f9-kserve-self-signed-certs\""
Sep 16 13:05:50.458404 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.458303    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e344f9\"/\"kube-root-ca.crt\""
Sep 16 13:05:50.464521 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.464500    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"]
Sep 16 13:05:50.564828 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.564791    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-dshm\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.565002 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.564835    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-home\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.565002 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.564853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-model-cache\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.565002 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.564960    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tmp-dir\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.565002 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.564983    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tls-certs\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.565160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.565003    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfhfb\" (UniqueName: \"kubernetes.io/projected/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-kube-api-access-cfhfb\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.666366 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.666334    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-dshm\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.666502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.666377    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-home\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.666502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.666395    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-model-cache\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.666502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.666443    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tmp-dir\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.666502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.666458    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tls-certs\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.666502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.666482    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfhfb\" (UniqueName: \"kubernetes.io/projected/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-kube-api-access-cfhfb\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.667264 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.667163    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-home\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.667264 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.667202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-model-cache\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.667513 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.667253    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tmp-dir\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.669974 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.669947    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-dshm\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.670276 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.670252    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tls-certs\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.676738 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.676711    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfhfb\" (UniqueName: \"kubernetes.io/projected/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-kube-api-access-cfhfb\") pod \"auth-test-model-e344f9-kserve-97bdd67-j2tx5\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") " pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:50.764707 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:50.764616    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:05:51.039778 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:51.039756    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"]
Sep 16 13:05:51.042097 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:05:51.042059    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4772fc3c_2f25_4908_a3e2_81d1aeb4f290.slice/crio-23c429cf194f497e7e6985c87f0c9a874ef479d3ed52234609d9b53604f7ad61 WatchSource:0}: Error finding container 23c429cf194f497e7e6985c87f0c9a874ef479d3ed52234609d9b53604f7ad61: Status 404 returned error can't find the container with id 23c429cf194f497e7e6985c87f0c9a874ef479d3ed52234609d9b53604f7ad61
Sep 16 13:05:51.235616 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:51.235582    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5" event={"ID":"4772fc3c-2f25-4908-a3e2-81d1aeb4f290","Type":"ContainerStarted","Data":"9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7"}
Sep 16 13:05:51.235773 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:51.235628    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5" event={"ID":"4772fc3c-2f25-4908-a3e2-81d1aeb4f290","Type":"ContainerStarted","Data":"23c429cf194f497e7e6985c87f0c9a874ef479d3ed52234609d9b53604f7ad61"}
Sep 16 13:05:51.255238 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:05:51.255188    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5" podStartSLOduration=1.255174041 podStartE2EDuration="1.255174041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:05:51.252899348 +0000 UTC m=+1653.336342395" watchObservedRunningTime="2026-09-16 13:05:51.255174041 +0000 UTC m=+1653.338617055"
Sep 16 13:06:00.765571 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:00.765531    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:06:00.765571 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:00.765576    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:06:00.780217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:00.780188    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:06:01.297957 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:01.297926    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:06:09.745530 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:09.745492    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:06:18.625137 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:18.625088    2559 scope.go:117] "RemoveContainer" containerID="6d92efbdbcf7c4e65ef3ea4da22b3ba93e3e8e1a4ad8d7227f73a5e4d82b5256"
Sep 16 13:06:18.633975 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:18.633954    2559 scope.go:117] "RemoveContainer" containerID="d786fa03a45324a13f6a6b7a66e06715e366091c00f54c81ad913be234c8feb4"
Sep 16 13:06:51.355905 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:51.355827    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:06:56.049531 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.049495    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"]
Sep 16 13:06:56.053218 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.053201    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.055550 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.055532    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-7d0256\"/\"distinct-2-w4-7d0256-kserve-self-signed-certs\""
Sep 16 13:06:56.065395 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.065372    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"]
Sep 16 13:06:56.179174 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.179142    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tmp-dir\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.179174 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.179177    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tls-certs\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.179414 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.179202    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-dshm\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.179414 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.179228    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pdkt\" (UniqueName: \"kubernetes.io/projected/f448ee9e-5531-4d39-8305-d0bfa1fe0961-kube-api-access-6pdkt\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.179414 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.179290    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-model-cache\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.179414 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.179360    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-home\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280292 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280246    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tmp-dir\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280292 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280294    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tls-certs\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-dshm\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280352    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pdkt\" (UniqueName: \"kubernetes.io/projected/f448ee9e-5531-4d39-8305-d0bfa1fe0961-kube-api-access-6pdkt\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280373    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-model-cache\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280504    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-home\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280755 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280720    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tmp-dir\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280806 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280787    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-model-cache\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.280868 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.280853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-home\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.282713 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.282696    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-dshm\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.283027 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.283012    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tls-certs\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.288690 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.288666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pdkt\" (UniqueName: \"kubernetes.io/projected/f448ee9e-5531-4d39-8305-d0bfa1fe0961-kube-api-access-6pdkt\") pod \"distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") " pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.362491 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.362425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:06:56.635838 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:56.635814    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"]
Sep 16 13:06:56.637754 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:06:56.637727    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf448ee9e_5531_4d39_8305_d0bfa1fe0961.slice/crio-46fcdeac2801c9c8cb4e0ffd608378e9b39a55bc5863f64163d5315da6ae8c44 WatchSource:0}: Error finding container 46fcdeac2801c9c8cb4e0ffd608378e9b39a55bc5863f64163d5315da6ae8c44: Status 404 returned error can't find the container with id 46fcdeac2801c9c8cb4e0ffd608378e9b39a55bc5863f64163d5315da6ae8c44
Sep 16 13:06:57.558497 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:57.558453    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7" event={"ID":"f448ee9e-5531-4d39-8305-d0bfa1fe0961","Type":"ContainerStarted","Data":"7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019"}
Sep 16 13:06:57.558882 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:57.558510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7" event={"ID":"f448ee9e-5531-4d39-8305-d0bfa1fe0961","Type":"ContainerStarted","Data":"46fcdeac2801c9c8cb4e0ffd608378e9b39a55bc5863f64163d5315da6ae8c44"}
Sep 16 13:06:57.575614 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:06:57.575558    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7" podStartSLOduration=1.575542427 podStartE2EDuration="1.575542427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:06:57.57450007 +0000 UTC m=+1719.657943100" watchObservedRunningTime="2026-09-16 13:06:57.575542427 +0000 UTC m=+1719.658985441"
Sep 16 13:07:06.362606 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:06.362576    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:07:06.363012 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:06.362624    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:07:06.380271 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:06.380242    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:07:06.613613 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:06.613533    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:07:17.118817 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.118783    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"]
Sep 16 13:07:17.123397 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.123375    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.125500 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.125481    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f0722f-serving-cert\""
Sep 16 13:07:17.134674 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.134653    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"]
Sep 16 13:07:17.174814 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.174784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-metrics-certs\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.174991 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.174945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.174991 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.174980    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f0672f-e1c3-496b-bf2a-3661964eaad6-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.175105 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.175009    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnd8\" (UniqueName: \"kubernetes.io/projected/72f0672f-e1c3-496b-bf2a-3661964eaad6-kube-api-access-8fnd8\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.276378 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.276312    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.276577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.276394    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f0672f-e1c3-496b-bf2a-3661964eaad6-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.276577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.276423    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnd8\" (UniqueName: \"kubernetes.io/projected/72f0672f-e1c3-496b-bf2a-3661964eaad6-kube-api-access-8fnd8\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.276577 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:07:17.276430    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-serving-cert: secret "maas-api-e2e-api-a-f0722f-serving-cert" not found
Sep 16 13:07:17.276577 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:07:17.276494    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls podName:72f0672f-e1c3-496b-bf2a-3661964eaad6 nodeName:}" failed. No retries permitted until 2026-09-16 13:07:17.77647733 +0000 UTC m=+1739.859920322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls") pod "maas-api-e2e-api-a-f0722f-59b4995f69-strls" (UID: "72f0672f-e1c3-496b-bf2a-3661964eaad6") : secret "maas-api-e2e-api-a-f0722f-serving-cert" not found
Sep 16 13:07:17.276577 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.276511    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-metrics-certs\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.277034 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.277005    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f0672f-e1c3-496b-bf2a-3661964eaad6-openshift-service-ca\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.279176 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.279144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-metrics-certs\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.286000 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.285976    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnd8\" (UniqueName: \"kubernetes.io/projected/72f0672f-e1c3-496b-bf2a-3661964eaad6-kube-api-access-8fnd8\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.782431 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.782400    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:17.789875 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:17.785859    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls\") pod \"maas-api-e2e-api-a-f0722f-59b4995f69-strls\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:18.036392 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:18.036295    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:18.154042 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:18.154010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:07:18.310744 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:18.310717    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"]
Sep 16 13:07:18.313736 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:07:18.313712    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f0672f_e1c3_496b_bf2a_3661964eaad6.slice/crio-cebbc7e0fdf803e57204e05551e6a0c493d8bb26077f83dd2a7a464af14ef66d WatchSource:0}: Error finding container cebbc7e0fdf803e57204e05551e6a0c493d8bb26077f83dd2a7a464af14ef66d: Status 404 returned error can't find the container with id cebbc7e0fdf803e57204e05551e6a0c493d8bb26077f83dd2a7a464af14ef66d
Sep 16 13:07:18.315103 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:18.315085    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:07:18.661807 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:18.661724    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls" event={"ID":"72f0672f-e1c3-496b-bf2a-3661964eaad6","Type":"ContainerStarted","Data":"cebbc7e0fdf803e57204e05551e6a0c493d8bb26077f83dd2a7a464af14ef66d"}
Sep 16 13:07:19.668195 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:19.668159    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls" event={"ID":"72f0672f-e1c3-496b-bf2a-3661964eaad6","Type":"ContainerStarted","Data":"692c88ca87f288d70ce4f0db6b07f8468700ca4e601f8c57332d9b93135013d3"}
Sep 16 13:07:19.668655 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:19.668292    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:19.687842 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:19.687798    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls" podStartSLOduration=2.151664837 podStartE2EDuration="2.687783432s" totalImagesPullingTime="536.118595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:19.685499325 +0000 UTC m=+1741.768942353" watchObservedRunningTime="2026-09-16 13:07:19.687783432 +0000 UTC m=+1741.771226483"
Sep 16 13:07:25.678443 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:25.678410    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:07:44.754665 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:44.754632    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:07:52.821998 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.821969    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"]
Sep 16 13:07:52.825760 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.825741    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:52.827816 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.827798    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-94f3e1-serving-cert\""
Sep 16 13:07:52.836034 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.836010    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"]
Sep 16 13:07:52.914870 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.914839    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5vjg\" (UniqueName: \"kubernetes.io/projected/f7e51ad0-92f0-4872-8333-4c14af833fcb-kube-api-access-q5vjg\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:52.915030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.914877    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-metrics-certs\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:52.915030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.914910    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e51ad0-92f0-4872-8333-4c14af833fcb-openshift-service-ca\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:52.915109 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:52.915031    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.015930 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.015891    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5vjg\" (UniqueName: \"kubernetes.io/projected/f7e51ad0-92f0-4872-8333-4c14af833fcb-kube-api-access-q5vjg\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.015930 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.015942    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-metrics-certs\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.016164 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.015973    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e51ad0-92f0-4872-8333-4c14af833fcb-openshift-service-ca\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.016164 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.016032    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.016729 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.016703    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e51ad0-92f0-4872-8333-4c14af833fcb-openshift-service-ca\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.018591 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.018564    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-maas-api-tls\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.018591 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.018583    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-metrics-certs\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.024002 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.023977    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5vjg\" (UniqueName: \"kubernetes.io/projected/f7e51ad0-92f0-4872-8333-4c14af833fcb-kube-api-access-q5vjg\") pod \"maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.137265 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.137183    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.409046 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.409023    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"]
Sep 16 13:07:53.411635 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:07:53.411606    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e51ad0_92f0_4872_8333_4c14af833fcb.slice/crio-9bbda57701be0786d963bfb2aed7cd3e091c080444564ba4fdfad9b0f0bce608 WatchSource:0}: Error finding container 9bbda57701be0786d963bfb2aed7cd3e091c080444564ba4fdfad9b0f0bce608: Status 404 returned error can't find the container with id 9bbda57701be0786d963bfb2aed7cd3e091c080444564ba4fdfad9b0f0bce608
Sep 16 13:07:53.846389 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.846349    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7" event={"ID":"f7e51ad0-92f0-4872-8333-4c14af833fcb","Type":"ContainerStarted","Data":"5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb"}
Sep 16 13:07:53.846750 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.846413    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7" event={"ID":"f7e51ad0-92f0-4872-8333-4c14af833fcb","Type":"ContainerStarted","Data":"9bbda57701be0786d963bfb2aed7cd3e091c080444564ba4fdfad9b0f0bce608"}
Sep 16 13:07:53.846750 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.846513    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:07:53.866007 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:53.865963    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7" podStartSLOduration=1.519914151 podStartE2EDuration="1.865947371s" totalImagesPullingTime="346.03322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:07:53.863949918 +0000 UTC m=+1775.947392964" watchObservedRunningTime="2026-09-16 13:07:53.865947371 +0000 UTC m=+1775.949390446"
Sep 16 13:07:59.857866 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:07:59.857839    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:08:18.645988 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:18.645955    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:08:35.934996 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:35.934955    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"]
Sep 16 13:08:35.935575 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:35.935203    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls" podUID="72f0672f-e1c3-496b-bf2a-3661964eaad6" containerName="maas-api" containerID="cri-o://692c88ca87f288d70ce4f0db6b07f8468700ca4e601f8c57332d9b93135013d3" gracePeriod=30
Sep 16 13:08:36.065794 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.065754    2559 generic.go:356] "Generic (PLEG): container finished" podID="72f0672f-e1c3-496b-bf2a-3661964eaad6" containerID="692c88ca87f288d70ce4f0db6b07f8468700ca4e601f8c57332d9b93135013d3" exitCode=0
Sep 16 13:08:36.065945 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.065826    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls" event={"ID":"72f0672f-e1c3-496b-bf2a-3661964eaad6","Type":"ContainerDied","Data":"692c88ca87f288d70ce4f0db6b07f8468700ca4e601f8c57332d9b93135013d3"}
Sep 16 13:08:36.193822 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.193747    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:08:36.230427 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.230394    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fnd8\" (UniqueName: \"kubernetes.io/projected/72f0672f-e1c3-496b-bf2a-3661964eaad6-kube-api-access-8fnd8\") pod \"72f0672f-e1c3-496b-bf2a-3661964eaad6\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") "
Sep 16 13:08:36.230427 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.230433    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f0672f-e1c3-496b-bf2a-3661964eaad6-openshift-service-ca\") pod \"72f0672f-e1c3-496b-bf2a-3661964eaad6\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") "
Sep 16 13:08:36.230668 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.230489    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls\") pod \"72f0672f-e1c3-496b-bf2a-3661964eaad6\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") "
Sep 16 13:08:36.230668 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.230606    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-metrics-certs\") pod \"72f0672f-e1c3-496b-bf2a-3661964eaad6\" (UID: \"72f0672f-e1c3-496b-bf2a-3661964eaad6\") "
Sep 16 13:08:36.232453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.231234    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f0672f-e1c3-496b-bf2a-3661964eaad6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "72f0672f-e1c3-496b-bf2a-3661964eaad6" (UID: "72f0672f-e1c3-496b-bf2a-3661964eaad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:36.232453 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.231588    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72f0672f-e1c3-496b-bf2a-3661964eaad6-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:36.233706 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.233648    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "72f0672f-e1c3-496b-bf2a-3661964eaad6" (UID: "72f0672f-e1c3-496b-bf2a-3661964eaad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:08:36.235429 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.234135    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0672f-e1c3-496b-bf2a-3661964eaad6-kube-api-access-8fnd8" (OuterVolumeSpecName: "kube-api-access-8fnd8") pod "72f0672f-e1c3-496b-bf2a-3661964eaad6" (UID: "72f0672f-e1c3-496b-bf2a-3661964eaad6"). InnerVolumeSpecName "kube-api-access-8fnd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:36.236364 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.235801    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "72f0672f-e1c3-496b-bf2a-3661964eaad6" (UID: "72f0672f-e1c3-496b-bf2a-3661964eaad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:08:36.332760 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.332730    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:36.332760 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.332757    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f0672f-e1c3-496b-bf2a-3661964eaad6-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:36.332951 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:36.332767    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fnd8\" (UniqueName: \"kubernetes.io/projected/72f0672f-e1c3-496b-bf2a-3661964eaad6-kube-api-access-8fnd8\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:37.072809 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:37.072772    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls" event={"ID":"72f0672f-e1c3-496b-bf2a-3661964eaad6","Type":"ContainerDied","Data":"cebbc7e0fdf803e57204e05551e6a0c493d8bb26077f83dd2a7a464af14ef66d"}
Sep 16 13:08:37.072809 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:37.072814    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"
Sep 16 13:08:37.073293 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:37.072834    2559 scope.go:117] "RemoveContainer" containerID="692c88ca87f288d70ce4f0db6b07f8468700ca4e601f8c57332d9b93135013d3"
Sep 16 13:08:37.094413 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:37.094385    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"]
Sep 16 13:08:37.098190 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:37.098160    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f0722f-59b4995f69-strls"]
Sep 16 13:08:38.412710 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:38.412678    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72f0672f-e1c3-496b-bf2a-3661964eaad6" path="/var/lib/kubelet/pods/72f0672f-e1c3-496b-bf2a-3661964eaad6/volumes"
Sep 16 13:08:43.964718 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:43.964682    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:08:59.119223 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.119188    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"]
Sep 16 13:08:59.119667 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.119423    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7" podUID="f7e51ad0-92f0-4872-8333-4c14af833fcb" containerName="maas-api" containerID="cri-o://5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb" gracePeriod=30
Sep 16 13:08:59.364717 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.364690    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:08:59.373658 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.373609    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e51ad0-92f0-4872-8333-4c14af833fcb-openshift-service-ca\") pod \"f7e51ad0-92f0-4872-8333-4c14af833fcb\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") "
Sep 16 13:08:59.373739 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.373728    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5vjg\" (UniqueName: \"kubernetes.io/projected/f7e51ad0-92f0-4872-8333-4c14af833fcb-kube-api-access-q5vjg\") pod \"f7e51ad0-92f0-4872-8333-4c14af833fcb\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") "
Sep 16 13:08:59.373784 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.373750    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-metrics-certs\") pod \"f7e51ad0-92f0-4872-8333-4c14af833fcb\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") "
Sep 16 13:08:59.373784 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.373772    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-maas-api-tls\") pod \"f7e51ad0-92f0-4872-8333-4c14af833fcb\" (UID: \"f7e51ad0-92f0-4872-8333-4c14af833fcb\") "
Sep 16 13:08:59.374242 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.373967    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e51ad0-92f0-4872-8333-4c14af833fcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7e51ad0-92f0-4872-8333-4c14af833fcb" (UID: "f7e51ad0-92f0-4872-8333-4c14af833fcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:08:59.374242 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.374112    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7e51ad0-92f0-4872-8333-4c14af833fcb-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:59.376027 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.375999    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e51ad0-92f0-4872-8333-4c14af833fcb-kube-api-access-q5vjg" (OuterVolumeSpecName: "kube-api-access-q5vjg") pod "f7e51ad0-92f0-4872-8333-4c14af833fcb" (UID: "f7e51ad0-92f0-4872-8333-4c14af833fcb"). InnerVolumeSpecName "kube-api-access-q5vjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:08:59.376125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.376061    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7e51ad0-92f0-4872-8333-4c14af833fcb" (UID: "f7e51ad0-92f0-4872-8333-4c14af833fcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:08:59.376125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.376081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7e51ad0-92f0-4872-8333-4c14af833fcb" (UID: "f7e51ad0-92f0-4872-8333-4c14af833fcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:08:59.474598 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.474566    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5vjg\" (UniqueName: \"kubernetes.io/projected/f7e51ad0-92f0-4872-8333-4c14af833fcb-kube-api-access-q5vjg\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:59.474598 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.474597    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:08:59.474598 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:08:59.474606    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7e51ad0-92f0-4872-8333-4c14af833fcb-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:00.190678 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.190640    2559 generic.go:356] "Generic (PLEG): container finished" podID="f7e51ad0-92f0-4872-8333-4c14af833fcb" containerID="5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb" exitCode=0
Sep 16 13:09:00.191146 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.190689    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"
Sep 16 13:09:00.191146 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.190738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7" event={"ID":"f7e51ad0-92f0-4872-8333-4c14af833fcb","Type":"ContainerDied","Data":"5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb"}
Sep 16 13:09:00.191146 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.190804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7" event={"ID":"f7e51ad0-92f0-4872-8333-4c14af833fcb","Type":"ContainerDied","Data":"9bbda57701be0786d963bfb2aed7cd3e091c080444564ba4fdfad9b0f0bce608"}
Sep 16 13:09:00.191146 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.190840    2559 scope.go:117] "RemoveContainer" containerID="5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb"
Sep 16 13:09:00.200509 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.200487    2559 scope.go:117] "RemoveContainer" containerID="5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb"
Sep 16 13:09:00.200811 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:00.200782    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb\": container with ID starting with 5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb not found: ID does not exist" containerID="5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb"
Sep 16 13:09:00.200917 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.200811    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb"} err="failed to get container status \"5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb\": rpc error: code = NotFound desc = could not find container \"5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb\": container with ID starting with 5eb7f5e5549968da49bf869c2dd991236f737abc98338f08d3bc31eb6a01c5bb not found: ID does not exist"
Sep 16 13:09:00.212375 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.212349    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"]
Sep 16 13:09:00.217133 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.217105    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-94f3e1-567688df4f-zbfc7"]
Sep 16 13:09:00.412477 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:00.412444    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e51ad0-92f0-4872-8333-4c14af833fcb" path="/var/lib/kubelet/pods/f7e51ad0-92f0-4872-8333-4c14af833fcb/volumes"
Sep 16 13:09:10.456136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:10.456099    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:09:21.588685 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.588648    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"]
Sep 16 13:09:21.589108 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.588967    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" podUID="aabb55e7-a3c3-449b-b00e-0efd9b766be1" containerName="maas-api" containerID="cri-o://0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce" gracePeriod=30
Sep 16 13:09:21.678603 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.678569    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"]
Sep 16 13:09:21.678861 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.678816    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh" podUID="24918870-b1b7-46a0-b9dc-eb6025c56491" containerName="payload-pre-processing" containerID="cri-o://8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1" gracePeriod=30
Sep 16 13:09:21.932771 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.932745    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:09:21.951071 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.951032    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"]
Sep 16 13:09:21.962136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.962113    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:09:21.995861 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.995829    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"]
Sep 16 13:09:21.998343 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.998285    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls\") pod \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") "
Sep 16 13:09:21.998541 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.998373    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smfs5\" (UniqueName: \"kubernetes.io/projected/aabb55e7-a3c3-449b-b00e-0efd9b766be1-kube-api-access-smfs5\") pod \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") "
Sep 16 13:09:21.998961 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.998760    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-metrics-certs\") pod \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") "
Sep 16 13:09:21.998961 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.998848    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabb55e7-a3c3-449b-b00e-0efd9b766be1-openshift-service-ca\") pod \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\" (UID: \"aabb55e7-a3c3-449b-b00e-0efd9b766be1\") "
Sep 16 13:09:21.999712 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:21.999679    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabb55e7-a3c3-449b-b00e-0efd9b766be1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aabb55e7-a3c3-449b-b00e-0efd9b766be1" (UID: "aabb55e7-a3c3-449b-b00e-0efd9b766be1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:22.002220 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.002178    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabb55e7-a3c3-449b-b00e-0efd9b766be1-kube-api-access-smfs5" (OuterVolumeSpecName: "kube-api-access-smfs5") pod "aabb55e7-a3c3-449b-b00e-0efd9b766be1" (UID: "aabb55e7-a3c3-449b-b00e-0efd9b766be1"). InnerVolumeSpecName "kube-api-access-smfs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:22.002466 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.002424    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aabb55e7-a3c3-449b-b00e-0efd9b766be1" (UID: "aabb55e7-a3c3-449b-b00e-0efd9b766be1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:22.002867 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.002846    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aabb55e7-a3c3-449b-b00e-0efd9b766be1" (UID: "aabb55e7-a3c3-449b-b00e-0efd9b766be1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:22.100563 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100469    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h5tm\" (UniqueName: \"kubernetes.io/projected/24918870-b1b7-46a0-b9dc-eb6025c56491-kube-api-access-4h5tm\") pod \"24918870-b1b7-46a0-b9dc-eb6025c56491\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") "
Sep 16 13:09:22.100563 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100510    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24918870-b1b7-46a0-b9dc-eb6025c56491-plugins-config-volume\") pod \"24918870-b1b7-46a0-b9dc-eb6025c56491\" (UID: \"24918870-b1b7-46a0-b9dc-eb6025c56491\") "
Sep 16 13:09:22.100758 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100719    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aabb55e7-a3c3-449b-b00e-0efd9b766be1-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.100758 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100729    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.100758 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100738    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smfs5\" (UniqueName: \"kubernetes.io/projected/aabb55e7-a3c3-449b-b00e-0efd9b766be1-kube-api-access-smfs5\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.100758 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100747    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aabb55e7-a3c3-449b-b00e-0efd9b766be1-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.100893 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.100867    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24918870-b1b7-46a0-b9dc-eb6025c56491-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24918870-b1b7-46a0-b9dc-eb6025c56491" (UID: "24918870-b1b7-46a0-b9dc-eb6025c56491"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:22.102743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.102723    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24918870-b1b7-46a0-b9dc-eb6025c56491-kube-api-access-4h5tm" (OuterVolumeSpecName: "kube-api-access-4h5tm") pod "24918870-b1b7-46a0-b9dc-eb6025c56491" (UID: "24918870-b1b7-46a0-b9dc-eb6025c56491"). InnerVolumeSpecName "kube-api-access-4h5tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:22.202345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.202283    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h5tm\" (UniqueName: \"kubernetes.io/projected/24918870-b1b7-46a0-b9dc-eb6025c56491-kube-api-access-4h5tm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.202345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.202314    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24918870-b1b7-46a0-b9dc-eb6025c56491-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:22.308425 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.308392    2559 generic.go:356] "Generic (PLEG): container finished" podID="24918870-b1b7-46a0-b9dc-eb6025c56491" containerID="8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1" exitCode=0
Sep 16 13:09:22.308637 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.308443    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"
Sep 16 13:09:22.308637 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.308480    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh" event={"ID":"24918870-b1b7-46a0-b9dc-eb6025c56491","Type":"ContainerDied","Data":"8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1"}
Sep 16 13:09:22.308637 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.308541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh" event={"ID":"24918870-b1b7-46a0-b9dc-eb6025c56491","Type":"ContainerDied","Data":"60c4447947ac7bdc96298f2f6a9f53a0c902a746d4d8233e341a77f206c2059f"}
Sep 16 13:09:22.308637 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.308564    2559 scope.go:117] "RemoveContainer" containerID="8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1"
Sep 16 13:09:22.309914 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.309890    2559 generic.go:356] "Generic (PLEG): container finished" podID="aabb55e7-a3c3-449b-b00e-0efd9b766be1" containerID="0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce" exitCode=0
Sep 16 13:09:22.310020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.309925    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" event={"ID":"aabb55e7-a3c3-449b-b00e-0efd9b766be1","Type":"ContainerDied","Data":"0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce"}
Sep 16 13:09:22.310020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.309949    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"
Sep 16 13:09:22.310020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.309963    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq" event={"ID":"aabb55e7-a3c3-449b-b00e-0efd9b766be1","Type":"ContainerDied","Data":"060219ec6f56e834cf0bc9ed979cae99399a278480da7993031f23758b8d461f"}
Sep 16 13:09:22.319267 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.319250    2559 scope.go:117] "RemoveContainer" containerID="8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1"
Sep 16 13:09:22.319626 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:22.319600    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1\": container with ID starting with 8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1 not found: ID does not exist" containerID="8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1"
Sep 16 13:09:22.319708 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.319637    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1"} err="failed to get container status \"8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1\": rpc error: code = NotFound desc = could not find container \"8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1\": container with ID starting with 8d31ac9b9e24c69941b404e42c921f873a40ca5bc2c41ebf97d7c9f6ad3c10d1 not found: ID does not exist"
Sep 16 13:09:22.319708 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.319659    2559 scope.go:117] "RemoveContainer" containerID="0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce"
Sep 16 13:09:22.327995 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.327974    2559 scope.go:117] "RemoveContainer" containerID="0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce"
Sep 16 13:09:22.328271 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:22.328252    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce\": container with ID starting with 0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce not found: ID does not exist" containerID="0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce"
Sep 16 13:09:22.328375 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.328280    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce"} err="failed to get container status \"0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce\": rpc error: code = NotFound desc = could not find container \"0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce\": container with ID starting with 0ade0b6fce5078cea67af158b1ff711cff615e83c30cca201e9495ce00b666ce not found: ID does not exist"
Sep 16 13:09:22.339859 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.339832    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"]
Sep 16 13:09:22.343402 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.343376    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-863d9d-6cd8dbcc58-dv4vh"]
Sep 16 13:09:22.365951 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.365831    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"]
Sep 16 13:09:22.368234 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.368208    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-863d9d-76956d777-x6vxq"]
Sep 16 13:09:22.411866 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.411837    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24918870-b1b7-46a0-b9dc-eb6025c56491" path="/var/lib/kubelet/pods/24918870-b1b7-46a0-b9dc-eb6025c56491/volumes"
Sep 16 13:09:22.412242 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:22.412229    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aabb55e7-a3c3-449b-b00e-0efd9b766be1" path="/var/lib/kubelet/pods/aabb55e7-a3c3-449b-b00e-0efd9b766be1/volumes"
Sep 16 13:09:27.924253 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924218    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7544-0cdf83"]
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924594    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e51ad0-92f0-4872-8333-4c14af833fcb" containerName="maas-api"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924605    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e51ad0-92f0-4872-8333-4c14af833fcb" containerName="maas-api"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924614    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabb55e7-a3c3-449b-b00e-0efd9b766be1" containerName="maas-api"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924621    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabb55e7-a3c3-449b-b00e-0efd9b766be1" containerName="maas-api"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924635    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24918870-b1b7-46a0-b9dc-eb6025c56491" containerName="payload-pre-processing"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924641    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="24918870-b1b7-46a0-b9dc-eb6025c56491" containerName="payload-pre-processing"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924659    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72f0672f-e1c3-496b-bf2a-3661964eaad6" containerName="maas-api"
Sep 16 13:09:27.924692 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924665    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72f0672f-e1c3-496b-bf2a-3661964eaad6" containerName="maas-api"
Sep 16 13:09:27.924980 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924737    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabb55e7-a3c3-449b-b00e-0efd9b766be1" containerName="maas-api"
Sep 16 13:09:27.924980 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924747    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72f0672f-e1c3-496b-bf2a-3661964eaad6" containerName="maas-api"
Sep 16 13:09:27.924980 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924756    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="24918870-b1b7-46a0-b9dc-eb6025c56491" containerName="payload-pre-processing"
Sep 16 13:09:27.924980 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.924763    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e51ad0-92f0-4872-8333-4c14af833fcb" containerName="maas-api"
Sep 16 13:09:27.929127 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.929110    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:27.932504 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.932484    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z9lzx\""
Sep 16 13:09:27.937021 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:27.936649    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-0cdf83"]
Sep 16 13:09:28.056797 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:28.056757    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d4nk\" (UniqueName: \"kubernetes.io/projected/8f186ceb-3f22-4835-b114-1447cc0fc79f-kube-api-access-5d4nk\") pod \"test-curl-7544-0cdf83\" (UID: \"8f186ceb-3f22-4835-b114-1447cc0fc79f\") " pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:28.157522 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:28.157481    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d4nk\" (UniqueName: \"kubernetes.io/projected/8f186ceb-3f22-4835-b114-1447cc0fc79f-kube-api-access-5d4nk\") pod \"test-curl-7544-0cdf83\" (UID: \"8f186ceb-3f22-4835-b114-1447cc0fc79f\") " pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:28.170119 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:28.170091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d4nk\" (UniqueName: \"kubernetes.io/projected/8f186ceb-3f22-4835-b114-1447cc0fc79f-kube-api-access-5d4nk\") pod \"test-curl-7544-0cdf83\" (UID: \"8f186ceb-3f22-4835-b114-1447cc0fc79f\") " pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:28.239714 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:28.239631    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:28.510186 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:28.510159    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-0cdf83"]
Sep 16 13:09:28.512970 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:09:28.512939    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f186ceb_3f22_4835_b114_1447cc0fc79f.slice/crio-a9ddfa516a8b8373df04e9c3705e761b1b666a9ede0f4a33f49a80ea379c6a2a WatchSource:0}: Error finding container a9ddfa516a8b8373df04e9c3705e761b1b666a9ede0f4a33f49a80ea379c6a2a: Status 404 returned error can't find the container with id a9ddfa516a8b8373df04e9c3705e761b1b666a9ede0f4a33f49a80ea379c6a2a
Sep 16 13:09:29.348208 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:29.348169    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-0cdf83" event={"ID":"8f186ceb-3f22-4835-b114-1447cc0fc79f","Type":"ContainerStarted","Data":"f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359"}
Sep 16 13:09:29.348623 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:29.348221    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-0cdf83" event={"ID":"8f186ceb-3f22-4835-b114-1447cc0fc79f","Type":"ContainerStarted","Data":"a9ddfa516a8b8373df04e9c3705e761b1b666a9ede0f4a33f49a80ea379c6a2a"}
Sep 16 13:09:29.364602 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:29.364555    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7544-0cdf83" podStartSLOduration=2.364533582 podStartE2EDuration="2.364533582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:29.363869814 +0000 UTC m=+1871.447312827" watchObservedRunningTime="2026-09-16 13:09:29.364533582 +0000 UTC m=+1871.447976596"
Sep 16 13:09:29.977079 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:29.977040    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7544-0cdf83"]
Sep 16 13:09:29.981775 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:29.981746    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7544-0cdf83"]
Sep 16 13:09:30.353658 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:30.353629    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7544-0cdf83" container="test-curl-7544-0cdf83"
Sep 16 13:09:31.358758 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:31.358717    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7544-0cdf83" podUID="8f186ceb-3f22-4835-b114-1447cc0fc79f" containerName="test-curl-7544-0cdf83" containerID="cri-o://f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359" gracePeriod=2
Sep 16 13:09:32.741867 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:32.741837    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"]
Sep 16 13:09:33.539933 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:33.539904    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:33.714621 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:33.714586    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d4nk\" (UniqueName: \"kubernetes.io/projected/8f186ceb-3f22-4835-b114-1447cc0fc79f-kube-api-access-5d4nk\") pod \"8f186ceb-3f22-4835-b114-1447cc0fc79f\" (UID: \"8f186ceb-3f22-4835-b114-1447cc0fc79f\") "
Sep 16 13:09:33.716845 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:33.716821    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f186ceb-3f22-4835-b114-1447cc0fc79f-kube-api-access-5d4nk" (OuterVolumeSpecName: "kube-api-access-5d4nk") pod "8f186ceb-3f22-4835-b114-1447cc0fc79f" (UID: "8f186ceb-3f22-4835-b114-1447cc0fc79f"). InnerVolumeSpecName "kube-api-access-5d4nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:33.816362 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:33.816303    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d4nk\" (UniqueName: \"kubernetes.io/projected/8f186ceb-3f22-4835-b114-1447cc0fc79f-kube-api-access-5d4nk\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:34.375916 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:34.375884    2559 generic.go:356] "Generic (PLEG): container finished" podID="8f186ceb-3f22-4835-b114-1447cc0fc79f" containerID="f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359" exitCode=137
Sep 16 13:09:34.376131 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:34.375933    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:34.376131 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:34.375979    2559 scope.go:117] "RemoveContainer" containerID="f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359"
Sep 16 13:09:34.385407 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:34.385389    2559 scope.go:117] "RemoveContainer" containerID="f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359"
Sep 16 13:09:34.385692 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:34.385674    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359\": container with ID starting with f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359 not found: ID does not exist" containerID="f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359"
Sep 16 13:09:34.385747 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:34.385702    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359"} err="failed to get container status \"f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359\": rpc error: code = NotFound desc = could not find container \"f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359\": container with ID starting with f6b2f08572b480d802340fe5eb7bf9e529c953f6b94b0623f22ff92beba43359 not found: ID does not exist"
Sep 16 13:09:34.386269 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:34.386250    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7544-0cdf83\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="8f186ceb-3f22-4835-b114-1447cc0fc79f" pod="opendatahub/test-curl-7544-0cdf83"
Sep 16 13:09:34.411973 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:34.411947    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f186ceb-3f22-4835-b114-1447cc0fc79f" path="/var/lib/kubelet/pods/8f186ceb-3f22-4835-b114-1447cc0fc79f/volumes"
Sep 16 13:09:36.251190 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:36.251149    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:09:36.971392 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:36.971342    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7" podUID="f448ee9e-5531-4d39-8305-d0bfa1fe0961" containerName="main" containerID="cri-o://7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019" gracePeriod=45
Sep 16 13:09:37.019598 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.019540    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6" podUID="095ee0f2-9efb-44ce-a883-751f66521957" containerName="main" containerID="cri-o://020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429" gracePeriod=45
Sep 16 13:09:37.276683 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.276658    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:09:37.299023 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.298990    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 13:09:37.351718 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.351686    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pdkt\" (UniqueName: \"kubernetes.io/projected/f448ee9e-5531-4d39-8305-d0bfa1fe0961-kube-api-access-6pdkt\") pod \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") "
Sep 16 13:09:37.351895 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.351747    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tls-certs\") pod \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") "
Sep 16 13:09:37.351895 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.351781    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n7td\" (UniqueName: \"kubernetes.io/projected/095ee0f2-9efb-44ce-a883-751f66521957-kube-api-access-6n7td\") pod \"095ee0f2-9efb-44ce-a883-751f66521957\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") "
Sep 16 13:09:37.351895 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.351803    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-tmp-dir\") pod \"095ee0f2-9efb-44ce-a883-751f66521957\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") "
Sep 16 13:09:37.351895 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.351848    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-dshm\") pod \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") "
Sep 16 13:09:37.352160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352134    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tmp-dir\") pod \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") "
Sep 16 13:09:37.352240 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352185    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-model-cache\") pod \"095ee0f2-9efb-44ce-a883-751f66521957\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") "
Sep 16 13:09:37.352240 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352212    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-model-cache\") pod \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") "
Sep 16 13:09:37.352360 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352247    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-home\") pod \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\" (UID: \"f448ee9e-5531-4d39-8305-d0bfa1fe0961\") "
Sep 16 13:09:37.352360 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352286    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-home\") pod \"095ee0f2-9efb-44ce-a883-751f66521957\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") "
Sep 16 13:09:37.352473 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-dshm\") pod \"095ee0f2-9efb-44ce-a883-751f66521957\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") "
Sep 16 13:09:37.352473 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352420    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095ee0f2-9efb-44ce-a883-751f66521957-tls-certs\") pod \"095ee0f2-9efb-44ce-a883-751f66521957\" (UID: \"095ee0f2-9efb-44ce-a883-751f66521957\") "
Sep 16 13:09:37.352734 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352622    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-home" (OuterVolumeSpecName: "home") pod "f448ee9e-5531-4d39-8305-d0bfa1fe0961" (UID: "f448ee9e-5531-4d39-8305-d0bfa1fe0961"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.353345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352800    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.353345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.352986    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "095ee0f2-9efb-44ce-a883-751f66521957" (UID: "095ee0f2-9efb-44ce-a883-751f66521957"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.353345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.353010    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f448ee9e-5531-4d39-8305-d0bfa1fe0961" (UID: "f448ee9e-5531-4d39-8305-d0bfa1fe0961"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.353345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.353183    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-model-cache" (OuterVolumeSpecName: "model-cache") pod "f448ee9e-5531-4d39-8305-d0bfa1fe0961" (UID: "f448ee9e-5531-4d39-8305-d0bfa1fe0961"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.353345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.353224    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-home" (OuterVolumeSpecName: "home") pod "095ee0f2-9efb-44ce-a883-751f66521957" (UID: "095ee0f2-9efb-44ce-a883-751f66521957"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.353345 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.353301    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-model-cache" (OuterVolumeSpecName: "model-cache") pod "095ee0f2-9efb-44ce-a883-751f66521957" (UID: "095ee0f2-9efb-44ce-a883-751f66521957"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.354601 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.354565    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f448ee9e-5531-4d39-8305-d0bfa1fe0961" (UID: "f448ee9e-5531-4d39-8305-d0bfa1fe0961"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:37.354780 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.354756    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095ee0f2-9efb-44ce-a883-751f66521957-kube-api-access-6n7td" (OuterVolumeSpecName: "kube-api-access-6n7td") pod "095ee0f2-9efb-44ce-a883-751f66521957" (UID: "095ee0f2-9efb-44ce-a883-751f66521957"). InnerVolumeSpecName "kube-api-access-6n7td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:37.354780 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.354770    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-dshm" (OuterVolumeSpecName: "dshm") pod "f448ee9e-5531-4d39-8305-d0bfa1fe0961" (UID: "f448ee9e-5531-4d39-8305-d0bfa1fe0961"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.355170 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.355146    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f448ee9e-5531-4d39-8305-d0bfa1fe0961-kube-api-access-6pdkt" (OuterVolumeSpecName: "kube-api-access-6pdkt") pod "f448ee9e-5531-4d39-8305-d0bfa1fe0961" (UID: "f448ee9e-5531-4d39-8305-d0bfa1fe0961"). InnerVolumeSpecName "kube-api-access-6pdkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:37.355982 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.355964    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-dshm" (OuterVolumeSpecName: "dshm") pod "095ee0f2-9efb-44ce-a883-751f66521957" (UID: "095ee0f2-9efb-44ce-a883-751f66521957"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:37.356275 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.356252    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095ee0f2-9efb-44ce-a883-751f66521957-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "095ee0f2-9efb-44ce-a883-751f66521957" (UID: "095ee0f2-9efb-44ce-a883-751f66521957"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:37.395930 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.395898    2559 generic.go:356] "Generic (PLEG): container finished" podID="f448ee9e-5531-4d39-8305-d0bfa1fe0961" containerID="7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019" exitCode=0
Sep 16 13:09:37.396117 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.395939    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"
Sep 16 13:09:37.396117 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.395984    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7" event={"ID":"f448ee9e-5531-4d39-8305-d0bfa1fe0961","Type":"ContainerDied","Data":"7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019"}
Sep 16 13:09:37.396117 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.396031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7" event={"ID":"f448ee9e-5531-4d39-8305-d0bfa1fe0961","Type":"ContainerDied","Data":"46fcdeac2801c9c8cb4e0ffd608378e9b39a55bc5863f64163d5315da6ae8c44"}
Sep 16 13:09:37.396117 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.396052    2559 scope.go:117] "RemoveContainer" containerID="7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019"
Sep 16 13:09:37.397409 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.397374    2559 generic.go:356] "Generic (PLEG): container finished" podID="095ee0f2-9efb-44ce-a883-751f66521957" containerID="020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429" exitCode=0
Sep 16 13:09:37.397517 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.397418    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"
Sep 16 13:09:37.397517 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.397478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6" event={"ID":"095ee0f2-9efb-44ce-a883-751f66521957","Type":"ContainerDied","Data":"020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429"}
Sep 16 13:09:37.397517 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.397506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6" event={"ID":"095ee0f2-9efb-44ce-a883-751f66521957","Type":"ContainerDied","Data":"8a7487d9ba5f4da642a6371a7f731e3776f62e929d6a82ce29598caebc074fbd"}
Sep 16 13:09:37.405667 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.405636    2559 scope.go:117] "RemoveContainer" containerID="7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019"
Sep 16 13:09:37.406016 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:37.405975    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019\": container with ID starting with 7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019 not found: ID does not exist" containerID="7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019"
Sep 16 13:09:37.406136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.406024    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019"} err="failed to get container status \"7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019\": rpc error: code = NotFound desc = could not find container \"7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019\": container with ID starting with 7687745a2e2d822e14683d8268307397bea8ccb015e91fdcba9e93fa8bca8019 not found: ID does not exist"
Sep 16 13:09:37.406136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.406054    2559 scope.go:117] "RemoveContainer" containerID="020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429"
Sep 16 13:09:37.414469 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.414452    2559 scope.go:117] "RemoveContainer" containerID="020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429"
Sep 16 13:09:37.414731 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:37.414715    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429\": container with ID starting with 020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429 not found: ID does not exist" containerID="020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429"
Sep 16 13:09:37.414780 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.414739    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429"} err="failed to get container status \"020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429\": rpc error: code = NotFound desc = could not find container \"020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429\": container with ID starting with 020ed56da95b2149dde26ddb45f4b396e45043067f7ed7ab20b05ee4ece8b429 not found: ID does not exist"
Sep 16 13:09:37.419060 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.419038    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"]
Sep 16 13:09:37.424594 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.424569    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/simulator-w4-7d0256-kserve-74467bbf59-xx7t6"]
Sep 16 13:09:37.434227 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.434201    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"]
Sep 16 13:09:37.437366 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.437348    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-7d0256/distinct-2-w4-7d0256-kserve-59bf979ffd-8mgd7"]
Sep 16 13:09:37.453938 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453906    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n7td\" (UniqueName: \"kubernetes.io/projected/095ee0f2-9efb-44ce-a883-751f66521957-kube-api-access-6n7td\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453938    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453949    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453957    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453965    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453974    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f448ee9e-5531-4d39-8305-d0bfa1fe0961-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453982    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453988    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095ee0f2-9efb-44ce-a883-751f66521957-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.453997    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095ee0f2-9efb-44ce-a883-751f66521957-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.454004    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pdkt\" (UniqueName: \"kubernetes.io/projected/f448ee9e-5531-4d39-8305-d0bfa1fe0961-kube-api-access-6pdkt\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:37.454273 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:37.454012    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f448ee9e-5531-4d39-8305-d0bfa1fe0961-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:38.412266 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:38.412241    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095ee0f2-9efb-44ce-a883-751f66521957" path="/var/lib/kubelet/pods/095ee0f2-9efb-44ce-a883-751f66521957/volumes"
Sep 16 13:09:38.412656 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:38.412642    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f448ee9e-5531-4d39-8305-d0bfa1fe0961" path="/var/lib/kubelet/pods/f448ee9e-5531-4d39-8305-d0bfa1fe0961/volumes"
Sep 16 13:09:39.862982 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:39.862908    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"]
Sep 16 13:09:39.863406 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:39.863143    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk" podUID="8c7412c7-8864-4562-b334-431a4ad12e97" containerName="maas-api" containerID="cri-o://0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652" gracePeriod=30
Sep 16 13:09:40.117790 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.117725    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:09:40.177899 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.177866    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-metrics-certs\") pod \"8c7412c7-8864-4562-b334-431a4ad12e97\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") "
Sep 16 13:09:40.178072 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.177943    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7412c7-8864-4562-b334-431a4ad12e97-openshift-service-ca\") pod \"8c7412c7-8864-4562-b334-431a4ad12e97\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") "
Sep 16 13:09:40.178072 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.177964    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z888z\" (UniqueName: \"kubernetes.io/projected/8c7412c7-8864-4562-b334-431a4ad12e97-kube-api-access-z888z\") pod \"8c7412c7-8864-4562-b334-431a4ad12e97\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") "
Sep 16 13:09:40.178072 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.177982    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-maas-api-tls\") pod \"8c7412c7-8864-4562-b334-431a4ad12e97\" (UID: \"8c7412c7-8864-4562-b334-431a4ad12e97\") "
Sep 16 13:09:40.178403 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.178373    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7412c7-8864-4562-b334-431a4ad12e97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8c7412c7-8864-4562-b334-431a4ad12e97" (UID: "8c7412c7-8864-4562-b334-431a4ad12e97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:09:40.180260 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.180233    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8c7412c7-8864-4562-b334-431a4ad12e97" (UID: "8c7412c7-8864-4562-b334-431a4ad12e97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:40.180382 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.180265    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8c7412c7-8864-4562-b334-431a4ad12e97" (UID: "8c7412c7-8864-4562-b334-431a4ad12e97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:40.180382 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.180351    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7412c7-8864-4562-b334-431a4ad12e97-kube-api-access-z888z" (OuterVolumeSpecName: "kube-api-access-z888z") pod "8c7412c7-8864-4562-b334-431a4ad12e97" (UID: "8c7412c7-8864-4562-b334-431a4ad12e97"). InnerVolumeSpecName "kube-api-access-z888z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:40.279686 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.279650    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7412c7-8864-4562-b334-431a4ad12e97-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.279686 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.279679    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z888z\" (UniqueName: \"kubernetes.io/projected/8c7412c7-8864-4562-b334-431a4ad12e97-kube-api-access-z888z\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.279686 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.279689    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.279909 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.279698    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7412c7-8864-4562-b334-431a4ad12e97-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:40.414802 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.414774    2559 generic.go:356] "Generic (PLEG): container finished" podID="8c7412c7-8864-4562-b334-431a4ad12e97" containerID="0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652" exitCode=0
Sep 16 13:09:40.414963 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.414837    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"
Sep 16 13:09:40.414963 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.414866    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk" event={"ID":"8c7412c7-8864-4562-b334-431a4ad12e97","Type":"ContainerDied","Data":"0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652"}
Sep 16 13:09:40.414963 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.414929    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk" event={"ID":"8c7412c7-8864-4562-b334-431a4ad12e97","Type":"ContainerDied","Data":"b16aef10a1f250eb582f5b50c39531d8ace17c8142648cb0005cf92698726ede"}
Sep 16 13:09:40.414963 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.414950    2559 scope.go:117] "RemoveContainer" containerID="0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652"
Sep 16 13:09:40.425179 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.425161    2559 scope.go:117] "RemoveContainer" containerID="0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652"
Sep 16 13:09:40.425494 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:40.425471    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652\": container with ID starting with 0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652 not found: ID does not exist" containerID="0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652"
Sep 16 13:09:40.425575 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.425503    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652"} err="failed to get container status \"0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652\": rpc error: code = NotFound desc = could not find container \"0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652\": container with ID starting with 0efd1a2af73f4e3001945abc0580f9a42fd73f25a8c65257a55d68ab3b13a652 not found: ID does not exist"
Sep 16 13:09:40.435782 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.435759    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"]
Sep 16 13:09:40.441594 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:40.441566    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e5926f-6d86cbbfb5-j7mlk"]
Sep 16 13:09:42.411737 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:42.411705    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c7412c7-8864-4562-b334-431a4ad12e97" path="/var/lib/kubelet/pods/8c7412c7-8864-4562-b334-431a4ad12e97/volumes"
Sep 16 13:09:44.932136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932100    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"]
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932468    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f448ee9e-5531-4d39-8305-d0bfa1fe0961" containerName="main"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932479    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f448ee9e-5531-4d39-8305-d0bfa1fe0961" containerName="main"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932491    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c7412c7-8864-4562-b334-431a4ad12e97" containerName="maas-api"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932496    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c7412c7-8864-4562-b334-431a4ad12e97" containerName="maas-api"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932517    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f186ceb-3f22-4835-b114-1447cc0fc79f" containerName="test-curl-7544-0cdf83"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932523    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f186ceb-3f22-4835-b114-1447cc0fc79f" containerName="test-curl-7544-0cdf83"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932531    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095ee0f2-9efb-44ce-a883-751f66521957" containerName="main"
Sep 16 13:09:44.932567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932537    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="095ee0f2-9efb-44ce-a883-751f66521957" containerName="main"
Sep 16 13:09:44.932818 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932588    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="095ee0f2-9efb-44ce-a883-751f66521957" containerName="main"
Sep 16 13:09:44.932818 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932596    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c7412c7-8864-4562-b334-431a4ad12e97" containerName="maas-api"
Sep 16 13:09:44.932818 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932603    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f448ee9e-5531-4d39-8305-d0bfa1fe0961" containerName="main"
Sep 16 13:09:44.932818 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.932609    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f186ceb-3f22-4835-b114-1447cc0fc79f" containerName="test-curl-7544-0cdf83"
Sep 16 13:09:44.936714 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.936697    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:44.939296 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.939275    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"kube-root-ca.crt\""
Sep 16 13:09:44.940245 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.940221    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"openshift-service-ca.crt\""
Sep 16 13:09:44.940356 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.940221    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"default-dockercfg-58kpl\""
Sep 16 13:09:44.940356 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.940229    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"sub-test-model-1ef95c-kserve-self-signed-certs\""
Sep 16 13:09:44.955614 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:44.948492    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"]
Sep 16 13:09:45.024930 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.024895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lw7\" (UniqueName: \"kubernetes.io/projected/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-kube-api-access-v8lw7\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.024930 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.024936    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-dshm\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.025125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.024954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-model-cache\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.025125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.025017    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tls-certs\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.025125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.025094    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tmp-dir\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.025234 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.025159    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-home\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126331 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126278    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lw7\" (UniqueName: \"kubernetes.io/projected/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-kube-api-access-v8lw7\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126498 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126336    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-dshm\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126498 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-model-cache\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126498 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126386    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tls-certs\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126498 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126408    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tmp-dir\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126498 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126470    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-home\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126829    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-model-cache\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.126855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126843    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tmp-dir\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.127061 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.126907    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-home\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.128822 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.128792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-dshm\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.129160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.129140    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tls-certs\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.135455 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.135431    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lw7\" (UniqueName: \"kubernetes.io/projected/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-kube-api-access-v8lw7\") pod \"sub-test-model-1ef95c-kserve-6455bb85d5-7st8d\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.246594 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.246502    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:45.535921 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:09:45.535892    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1dbd8ff_d6e7_4280_af73_c1456fb3f3f3.slice/crio-d9b547c34e88ba9b318f6398b63c378e14d445c4af8f18a6860fcc221aa9efdb WatchSource:0}: Error finding container d9b547c34e88ba9b318f6398b63c378e14d445c4af8f18a6860fcc221aa9efdb: Status 404 returned error can't find the container with id d9b547c34e88ba9b318f6398b63c378e14d445c4af8f18a6860fcc221aa9efdb
Sep 16 13:09:45.537743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:45.537720    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"]
Sep 16 13:09:46.445078 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:46.445043    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d" event={"ID":"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3","Type":"ContainerStarted","Data":"0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33"}
Sep 16 13:09:46.445078 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:46.445082    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d" event={"ID":"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3","Type":"ContainerStarted","Data":"d9b547c34e88ba9b318f6398b63c378e14d445c4af8f18a6860fcc221aa9efdb"}
Sep 16 13:09:46.464791 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:46.464746    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d" podStartSLOduration=2.464729675 podStartE2EDuration="2.464729675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:09:46.463058297 +0000 UTC m=+1888.546501312" watchObservedRunningTime="2026-09-16 13:09:46.464729675 +0000 UTC m=+1888.548172688"
Sep 16 13:09:47.761913 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:47.761878    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5" podUID="4772fc3c-2f25-4908-a3e2-81d1aeb4f290" containerName="main" containerID="cri-o://9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7" gracePeriod=45
Sep 16 13:09:48.012154 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.012084    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:09:48.056487 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056448    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-model-cache\") pod \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") "
Sep 16 13:09:48.056487 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056480    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfhfb\" (UniqueName: \"kubernetes.io/projected/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-kube-api-access-cfhfb\") pod \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") "
Sep 16 13:09:48.056750 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056515    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-dshm\") pod \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") "
Sep 16 13:09:48.056750 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056549    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tmp-dir\") pod \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") "
Sep 16 13:09:48.056750 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056612    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tls-certs\") pod \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") "
Sep 16 13:09:48.056750 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056674    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-home\") pod \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\" (UID: \"4772fc3c-2f25-4908-a3e2-81d1aeb4f290\") "
Sep 16 13:09:48.056971 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056777    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-model-cache" (OuterVolumeSpecName: "model-cache") pod "4772fc3c-2f25-4908-a3e2-81d1aeb4f290" (UID: "4772fc3c-2f25-4908-a3e2-81d1aeb4f290"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:48.056971 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056948    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4772fc3c-2f25-4908-a3e2-81d1aeb4f290" (UID: "4772fc3c-2f25-4908-a3e2-81d1aeb4f290"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:48.057082 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.056992    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-home" (OuterVolumeSpecName: "home") pod "4772fc3c-2f25-4908-a3e2-81d1aeb4f290" (UID: "4772fc3c-2f25-4908-a3e2-81d1aeb4f290"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:48.057133 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.057114    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:48.057176 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.057140    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:48.057176 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.057153    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:48.058943 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.058918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-dshm" (OuterVolumeSpecName: "dshm") pod "4772fc3c-2f25-4908-a3e2-81d1aeb4f290" (UID: "4772fc3c-2f25-4908-a3e2-81d1aeb4f290"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:09:48.059050 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.058956    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4772fc3c-2f25-4908-a3e2-81d1aeb4f290" (UID: "4772fc3c-2f25-4908-a3e2-81d1aeb4f290"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:09:48.059050 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.058964    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-kube-api-access-cfhfb" (OuterVolumeSpecName: "kube-api-access-cfhfb") pod "4772fc3c-2f25-4908-a3e2-81d1aeb4f290" (UID: "4772fc3c-2f25-4908-a3e2-81d1aeb4f290"). InnerVolumeSpecName "kube-api-access-cfhfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:09:48.158523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.158489    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfhfb\" (UniqueName: \"kubernetes.io/projected/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-kube-api-access-cfhfb\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:48.158523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.158517    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:48.158523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.158528    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4772fc3c-2f25-4908-a3e2-81d1aeb4f290-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:09:48.457649 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.457618    2559 generic.go:356] "Generic (PLEG): container finished" podID="4772fc3c-2f25-4908-a3e2-81d1aeb4f290" containerID="9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7" exitCode=0
Sep 16 13:09:48.457855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.457671    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"
Sep 16 13:09:48.457855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.457681    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5" event={"ID":"4772fc3c-2f25-4908-a3e2-81d1aeb4f290","Type":"ContainerDied","Data":"9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7"}
Sep 16 13:09:48.457855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.457802    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5" event={"ID":"4772fc3c-2f25-4908-a3e2-81d1aeb4f290","Type":"ContainerDied","Data":"23c429cf194f497e7e6985c87f0c9a874ef479d3ed52234609d9b53604f7ad61"}
Sep 16 13:09:48.457855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.457825    2559 scope.go:117] "RemoveContainer" containerID="9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7"
Sep 16 13:09:48.466828 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.466809    2559 scope.go:117] "RemoveContainer" containerID="9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7"
Sep 16 13:09:48.467108 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:09:48.467089    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7\": container with ID starting with 9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7 not found: ID does not exist" containerID="9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7"
Sep 16 13:09:48.467152 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.467117    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7"} err="failed to get container status \"9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7\": rpc error: code = NotFound desc = could not find container \"9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7\": container with ID starting with 9dcfcbda089f15eeea73625651ea67a4d5c59226e3a668236601b273b81510d7 not found: ID does not exist"
Sep 16 13:09:48.473967 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.473942    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"]
Sep 16 13:09:48.479594 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:48.479569    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e344f9/auth-test-model-e344f9-kserve-97bdd67-j2tx5"]
Sep 16 13:09:50.412564 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:50.412533    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4772fc3c-2f25-4908-a3e2-81d1aeb4f290" path="/var/lib/kubelet/pods/4772fc3c-2f25-4908-a3e2-81d1aeb4f290/volumes"
Sep 16 13:09:55.246950 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:55.246911    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:55.246950 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:55.246971    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:55.259617 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:55.259589    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:55.505456 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:55.505378    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:09:59.666701 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:09:59.666662    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:10:15.557412 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:15.557378    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:10:30.860926 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:30.860889    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:10:41.469021 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.468989    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7544-683288"]
Sep 16 13:10:41.469410 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.469350    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4772fc3c-2f25-4908-a3e2-81d1aeb4f290" containerName="main"
Sep 16 13:10:41.469410 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.469362    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4772fc3c-2f25-4908-a3e2-81d1aeb4f290" containerName="main"
Sep 16 13:10:41.469489 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.469436    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4772fc3c-2f25-4908-a3e2-81d1aeb4f290" containerName="main"
Sep 16 13:10:41.472572 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.472557    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:41.475502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.475482    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z9lzx\""
Sep 16 13:10:41.480084 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.480035    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-683288"]
Sep 16 13:10:41.569612 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.569572    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xcx\" (UniqueName: \"kubernetes.io/projected/9914d58b-b1b6-4060-84af-e506b98997c1-kube-api-access-k9xcx\") pod \"test-curl-7544-683288\" (UID: \"9914d58b-b1b6-4060-84af-e506b98997c1\") " pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:41.670688 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.670654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xcx\" (UniqueName: \"kubernetes.io/projected/9914d58b-b1b6-4060-84af-e506b98997c1-kube-api-access-k9xcx\") pod \"test-curl-7544-683288\" (UID: \"9914d58b-b1b6-4060-84af-e506b98997c1\") " pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:41.680491 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.680459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xcx\" (UniqueName: \"kubernetes.io/projected/9914d58b-b1b6-4060-84af-e506b98997c1-kube-api-access-k9xcx\") pod \"test-curl-7544-683288\" (UID: \"9914d58b-b1b6-4060-84af-e506b98997c1\") " pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:41.783385 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:41.783296    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:42.052846 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:42.052823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-683288"]
Sep 16 13:10:42.055147 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:10:42.055112    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9914d58b_b1b6_4060_84af_e506b98997c1.slice/crio-ca1df001e08561bba654a3f89b596d417ada2b9670758ad780c4f18b8dd9bec0 WatchSource:0}: Error finding container ca1df001e08561bba654a3f89b596d417ada2b9670758ad780c4f18b8dd9bec0: Status 404 returned error can't find the container with id ca1df001e08561bba654a3f89b596d417ada2b9670758ad780c4f18b8dd9bec0
Sep 16 13:10:42.727335 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:42.727295    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-683288" event={"ID":"9914d58b-b1b6-4060-84af-e506b98997c1","Type":"ContainerStarted","Data":"e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a"}
Sep 16 13:10:42.727715 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:42.727352    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-683288" event={"ID":"9914d58b-b1b6-4060-84af-e506b98997c1","Type":"ContainerStarted","Data":"ca1df001e08561bba654a3f89b596d417ada2b9670758ad780c4f18b8dd9bec0"}
Sep 16 13:10:42.743223 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:42.743175    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7544-683288" podStartSLOduration=1.743160691 podStartE2EDuration="1.743160691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:42.741420377 +0000 UTC m=+1944.824863392" watchObservedRunningTime="2026-09-16 13:10:42.743160691 +0000 UTC m=+1944.826603751"
Sep 16 13:10:43.323517 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.323478    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7544-683288"]
Sep 16 13:10:43.327275 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.327246    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7544-683288"]
Sep 16 13:10:43.442122 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.442085    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7544-2f9974"]
Sep 16 13:10:43.442481 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.442467    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" containerName="test-curl-7544-683288"
Sep 16 13:10:43.442534 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.442482    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" containerName="test-curl-7544-683288"
Sep 16 13:10:43.442582 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.442573    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" containerName="test-curl-7544-683288"
Sep 16 13:10:43.445763 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.445748    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:43.450495 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.450468    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-2f9974"]
Sep 16 13:10:43.589060 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.589030    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq5n9\" (UniqueName: \"kubernetes.io/projected/179a8977-69f6-48ec-b4fc-41911ed93afa-kube-api-access-nq5n9\") pod \"test-curl-7544-2f9974\" (UID: \"179a8977-69f6-48ec-b4fc-41911ed93afa\") " pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:43.690441 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.690406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq5n9\" (UniqueName: \"kubernetes.io/projected/179a8977-69f6-48ec-b4fc-41911ed93afa-kube-api-access-nq5n9\") pod \"test-curl-7544-2f9974\" (UID: \"179a8977-69f6-48ec-b4fc-41911ed93afa\") " pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:43.699397 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.699312    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq5n9\" (UniqueName: \"kubernetes.io/projected/179a8977-69f6-48ec-b4fc-41911ed93afa-kube-api-access-nq5n9\") pod \"test-curl-7544-2f9974\" (UID: \"179a8977-69f6-48ec-b4fc-41911ed93afa\") " pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:43.732739 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:43.732707    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7544-683288" container="test-curl-7544-683288"
Sep 16 13:10:43.755684 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:43.755658    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:44.024018 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:44.023939    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-2f9974"]
Sep 16 13:10:44.027737 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:10:44.027706    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod179a8977_69f6_48ec_b4fc_41911ed93afa.slice/crio-e27072dd005d6fb830c09a364b4c3d9c38dc5bae71eec6600ea0a42c2682e387 WatchSource:0}: Error finding container e27072dd005d6fb830c09a364b4c3d9c38dc5bae71eec6600ea0a42c2682e387: Status 404 returned error can't find the container with id e27072dd005d6fb830c09a364b4c3d9c38dc5bae71eec6600ea0a42c2682e387
Sep 16 13:10:44.738990 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:44.738956    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-2f9974" event={"ID":"179a8977-69f6-48ec-b4fc-41911ed93afa","Type":"ContainerStarted","Data":"7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34"}
Sep 16 13:10:44.739476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:44.738997    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-2f9974" event={"ID":"179a8977-69f6-48ec-b4fc-41911ed93afa","Type":"ContainerStarted","Data":"e27072dd005d6fb830c09a364b4c3d9c38dc5bae71eec6600ea0a42c2682e387"}
Sep 16 13:10:44.739476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:44.739148    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7544-683288" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" containerName="test-curl-7544-683288" containerID="cri-o://e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a" gracePeriod=2
Sep 16 13:10:44.754199 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:44.754152    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7544-2f9974" podStartSLOduration=1.7541366470000002 podStartE2EDuration="1.754136647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:44.752075622 +0000 UTC m=+1946.835518637" watchObservedRunningTime="2026-09-16 13:10:44.754136647 +0000 UTC m=+1946.837579661"
Sep 16 13:10:45.330370 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:45.330333    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7544-2f9974"]
Sep 16 13:10:45.334471 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:45.334441    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7544-2f9974"]
Sep 16 13:10:45.743707 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:45.743681    2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7544-2f9974" container="test-curl-7544-2f9974"
Sep 16 13:10:46.746970 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:46.746943    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:10:46.749107 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:46.749077    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7544-2f9974" podUID="179a8977-69f6-48ec-b4fc-41911ed93afa" containerName="test-curl-7544-2f9974" containerID="cri-o://7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34" gracePeriod=2
Sep 16 13:10:46.887970 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:46.887943    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:47.022398 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.022298    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9xcx\" (UniqueName: \"kubernetes.io/projected/9914d58b-b1b6-4060-84af-e506b98997c1-kube-api-access-k9xcx\") pod \"9914d58b-b1b6-4060-84af-e506b98997c1\" (UID: \"9914d58b-b1b6-4060-84af-e506b98997c1\") "
Sep 16 13:10:47.024503 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.024473    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9914d58b-b1b6-4060-84af-e506b98997c1-kube-api-access-k9xcx" (OuterVolumeSpecName: "kube-api-access-k9xcx") pod "9914d58b-b1b6-4060-84af-e506b98997c1" (UID: "9914d58b-b1b6-4060-84af-e506b98997c1"). InnerVolumeSpecName "kube-api-access-k9xcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:10:47.123724 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.123684    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xcx\" (UniqueName: \"kubernetes.io/projected/9914d58b-b1b6-4060-84af-e506b98997c1-kube-api-access-k9xcx\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:10:47.319397 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.319295    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"]
Sep 16 13:10:47.754567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.754526    2559 generic.go:356] "Generic (PLEG): container finished" podID="9914d58b-b1b6-4060-84af-e506b98997c1" containerID="e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a" exitCode=137
Sep 16 13:10:47.754567 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.754569    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:47.755007 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.754619    2559 scope.go:117] "RemoveContainer" containerID="e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a"
Sep 16 13:10:47.763817 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.763802    2559 scope.go:117] "RemoveContainer" containerID="e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a"
Sep 16 13:10:47.764080 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:47.764062    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a\": container with ID starting with e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a not found: ID does not exist" containerID="e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a"
Sep 16 13:10:47.764124 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:47.764091    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a"} err="failed to get container status \"e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a\": rpc error: code = NotFound desc = could not find container \"e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a\": container with ID starting with e0f1cbeb548157d766208249c0afe42d4afec5d473c8db189608bf014af55e2a not found: ID does not exist"
Sep 16 13:10:47.765450 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:47.765427    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7544-683288\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:48.409585 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:48.409545    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7544-683288\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" pod="opendatahub/test-curl-7544-683288"
Sep 16 13:10:48.411272 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:48.411249    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9914d58b-b1b6-4060-84af-e506b98997c1" path="/var/lib/kubelet/pods/9914d58b-b1b6-4060-84af-e506b98997c1/volumes"
Sep 16 13:10:48.908291 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:48.908263    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:49.041638 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.041538    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nq5n9\" (UniqueName: \"kubernetes.io/projected/179a8977-69f6-48ec-b4fc-41911ed93afa-kube-api-access-nq5n9\") pod \"179a8977-69f6-48ec-b4fc-41911ed93afa\" (UID: \"179a8977-69f6-48ec-b4fc-41911ed93afa\") "
Sep 16 13:10:49.043736 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.043702    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179a8977-69f6-48ec-b4fc-41911ed93afa-kube-api-access-nq5n9" (OuterVolumeSpecName: "kube-api-access-nq5n9") pod "179a8977-69f6-48ec-b4fc-41911ed93afa" (UID: "179a8977-69f6-48ec-b4fc-41911ed93afa"). InnerVolumeSpecName "kube-api-access-nq5n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:10:49.142929 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.142894    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq5n9\" (UniqueName: \"kubernetes.io/projected/179a8977-69f6-48ec-b4fc-41911ed93afa-kube-api-access-nq5n9\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:10:49.766371 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.766313    2559 generic.go:356] "Generic (PLEG): container finished" podID="179a8977-69f6-48ec-b4fc-41911ed93afa" containerID="7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34" exitCode=137
Sep 16 13:10:49.766371 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.766371    2559 scope.go:117] "RemoveContainer" containerID="7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34"
Sep 16 13:10:49.766596 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.766377    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:49.775213 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.775197    2559 scope.go:117] "RemoveContainer" containerID="7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34"
Sep 16 13:10:49.775507 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:49.775487    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34\": container with ID starting with 7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34 not found: ID does not exist" containerID="7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34"
Sep 16 13:10:49.775587 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:49.775513    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34"} err="failed to get container status \"7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34\": rpc error: code = NotFound desc = could not find container \"7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34\": container with ID starting with 7477a451773f101dec4686f6828f91339c7841f144f406ede193656be399da34 not found: ID does not exist"
Sep 16 13:10:49.776096 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:10:49.776073    2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7544-2f9974\" is forbidden: User \"system:node:ip-10-0-143-72.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-72.ec2.internal' and this object" podUID="179a8977-69f6-48ec-b4fc-41911ed93afa" pod="opendatahub/test-curl-7544-2f9974"
Sep 16 13:10:50.417919 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:50.417883    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179a8977-69f6-48ec-b4fc-41911ed93afa" path="/var/lib/kubelet/pods/179a8977-69f6-48ec-b4fc-41911ed93afa/volumes"
Sep 16 13:10:55.648532 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.648500    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"]
Sep 16 13:10:55.648889 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.648879    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179a8977-69f6-48ec-b4fc-41911ed93afa" containerName="test-curl-7544-2f9974"
Sep 16 13:10:55.648934 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.648891    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="179a8977-69f6-48ec-b4fc-41911ed93afa" containerName="test-curl-7544-2f9974"
Sep 16 13:10:55.648996 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.648985    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="179a8977-69f6-48ec-b4fc-41911ed93afa" containerName="test-curl-7544-2f9974"
Sep 16 13:10:55.652043 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.652025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.654359 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.654340    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1ef95c\"/\"rate-test-model-1ef95c-kserve-self-signed-certs\""
Sep 16 13:10:55.660018 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.659901    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"]
Sep 16 13:10:55.705636 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.705608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-tmp-dir\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.705636 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.705639    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6rbt\" (UniqueName: \"kubernetes.io/projected/6cb12616-fa9a-465b-8de4-121412078cb4-kube-api-access-b6rbt\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.705847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.705674    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb12616-fa9a-465b-8de4-121412078cb4-tls-certs\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.705847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.705695    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-model-cache\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.705847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.705712    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-home\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.705847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.705729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-dshm\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806359 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-dshm\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806559 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806447    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-tmp-dir\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806559 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6rbt\" (UniqueName: \"kubernetes.io/projected/6cb12616-fa9a-465b-8de4-121412078cb4-kube-api-access-b6rbt\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806559 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb12616-fa9a-465b-8de4-121412078cb4-tls-certs\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806735 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806560    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-model-cache\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806735 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806681    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-home\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806848 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-tmp-dir\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.806904 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.806877    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-model-cache\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.807095 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.807015    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-home\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.808700 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.808675    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-dshm\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.809110 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.809091    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb12616-fa9a-465b-8de4-121412078cb4-tls-certs\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.814292 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.814272    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6rbt\" (UniqueName: \"kubernetes.io/projected/6cb12616-fa9a-465b-8de4-121412078cb4-kube-api-access-b6rbt\") pod \"rate-test-model-1ef95c-kserve-7596487f79-77fxh\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") " pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:55.963550 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:55.963512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:10:56.234831 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:56.234740    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"]
Sep 16 13:10:56.237185 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:10:56.237160    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cb12616_fa9a_465b_8de4_121412078cb4.slice/crio-87d7b3678314cde6652553f000fb7db876336207ad1a60eed9ac29c1efa02411 WatchSource:0}: Error finding container 87d7b3678314cde6652553f000fb7db876336207ad1a60eed9ac29c1efa02411: Status 404 returned error can't find the container with id 87d7b3678314cde6652553f000fb7db876336207ad1a60eed9ac29c1efa02411
Sep 16 13:10:56.808370 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:56.808336    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh" event={"ID":"6cb12616-fa9a-465b-8de4-121412078cb4","Type":"ContainerStarted","Data":"c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473"}
Sep 16 13:10:56.808370 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:56.808378    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh" event={"ID":"6cb12616-fa9a-465b-8de4-121412078cb4","Type":"ContainerStarted","Data":"87d7b3678314cde6652553f000fb7db876336207ad1a60eed9ac29c1efa02411"}
Sep 16 13:10:56.826566 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:56.826517    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh" podStartSLOduration=1.8265018629999998 podStartE2EDuration="1.826501863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:10:56.824808136 +0000 UTC m=+1958.908251151" watchObservedRunningTime="2026-09-16 13:10:56.826501863 +0000 UTC m=+1958.909944876"
Sep 16 13:10:58.447815 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:10:58.447786    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:11:02.344299 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.344257    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d" podUID="f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" containerName="main" containerID="cri-o://0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33" gracePeriod=45
Sep 16 13:11:02.598464 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.598401    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:11:02.677478 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677435    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-dshm\") pod \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") "
Sep 16 13:11:02.677685 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677537    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8lw7\" (UniqueName: \"kubernetes.io/projected/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-kube-api-access-v8lw7\") pod \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") "
Sep 16 13:11:02.677685 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677564    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-home\") pod \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") "
Sep 16 13:11:02.677685 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677594    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-model-cache\") pod \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") "
Sep 16 13:11:02.677685 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677624    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tmp-dir\") pod \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") "
Sep 16 13:11:02.677911 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677719    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tls-certs\") pod \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\" (UID: \"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3\") "
Sep 16 13:11:02.677911 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677844    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-home" (OuterVolumeSpecName: "home") pod "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" (UID: "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:02.677911 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.677858    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" (UID: "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:02.678101 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.678074    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" (UID: "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:02.678161 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.678143    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:11:02.678208 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.678168    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:11:02.680086 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.680051    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-kube-api-access-v8lw7" (OuterVolumeSpecName: "kube-api-access-v8lw7") pod "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" (UID: "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3"). InnerVolumeSpecName "kube-api-access-v8lw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:11:02.680086 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.680069    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" (UID: "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:11:02.680086 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.680062    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-dshm" (OuterVolumeSpecName: "dshm") pod "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" (UID: "f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:11:02.779476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.779440    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8lw7\" (UniqueName: \"kubernetes.io/projected/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-kube-api-access-v8lw7\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:11:02.779476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.779470    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:11:02.779476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.779479    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:11:02.779476 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.779487    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:11:02.847307 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.847265    2559 generic.go:356] "Generic (PLEG): container finished" podID="f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" containerID="0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33" exitCode=0
Sep 16 13:11:02.847523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.847354    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"
Sep 16 13:11:02.847523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.847354    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d" event={"ID":"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3","Type":"ContainerDied","Data":"0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33"}
Sep 16 13:11:02.847523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.847414    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d" event={"ID":"f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3","Type":"ContainerDied","Data":"d9b547c34e88ba9b318f6398b63c378e14d445c4af8f18a6860fcc221aa9efdb"}
Sep 16 13:11:02.847523 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.847436    2559 scope.go:117] "RemoveContainer" containerID="0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33"
Sep 16 13:11:02.856951 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.856922    2559 scope.go:117] "RemoveContainer" containerID="0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33"
Sep 16 13:11:02.857228 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:11:02.857208    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33\": container with ID starting with 0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33 not found: ID does not exist" containerID="0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33"
Sep 16 13:11:02.857297 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.857237    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33"} err="failed to get container status \"0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33\": rpc error: code = NotFound desc = could not find container \"0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33\": container with ID starting with 0702ce82bdfe526fbef07eff377220cebf42f3cfd3d8db870a030f6fc0928f33 not found: ID does not exist"
Sep 16 13:11:02.867784 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.867755    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"]
Sep 16 13:11:02.871074 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:02.871052    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/sub-test-model-1ef95c-kserve-6455bb85d5-7st8d"]
Sep 16 13:11:04.415878 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:04.415845    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" path="/var/lib/kubelet/pods/f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3/volumes"
Sep 16 13:11:05.964178 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:05.964126    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:11:05.964178 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:05.964187    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:11:05.977562 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:05.977536    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:11:06.881204 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:06.881172    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:11:30.847229 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:11:30.847192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:12:01.552543 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:01.552508    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:12:11.144762 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:11.144709    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:12:17.680192 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:17.680158    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"]
Sep 16 13:12:21.649849 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:21.649816    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:12:24.021814 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.021782    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"]
Sep 16 13:12:24.022217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.022202    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" containerName="main"
Sep 16 13:12:24.022262 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.022218    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" containerName="main"
Sep 16 13:12:24.022297 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.022282    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1dbd8ff-d6e7-4280-af73-c1456fb3f3f3" containerName="main"
Sep 16 13:12:24.025293 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.025272    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.027441 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.027404    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a17de2-serving-cert\""
Sep 16 13:12:24.036888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.036868    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"]
Sep 16 13:12:24.075696 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.075662    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwtb4\" (UniqueName: \"kubernetes.io/projected/cb354199-e405-4f1f-a4ea-a95d238852bd-kube-api-access-pwtb4\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.075844 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.075731    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.075844 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.075801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb354199-e405-4f1f-a4ea-a95d238852bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.075926 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.075845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-metrics-certs\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.176446 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.176397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwtb4\" (UniqueName: \"kubernetes.io/projected/cb354199-e405-4f1f-a4ea-a95d238852bd-kube-api-access-pwtb4\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.176623 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.176469    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.176623 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.176500    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb354199-e405-4f1f-a4ea-a95d238852bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.176623 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.176525    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-metrics-certs\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.176746 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:12:24.176618    2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-serving-cert: secret "maas-api-e2e-ipp-a-a17de2-serving-cert" not found
Sep 16 13:12:24.176746 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:12:24.176704    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls podName:cb354199-e405-4f1f-a4ea-a95d238852bd nodeName:}" failed. No retries permitted until 2026-09-16 13:12:24.676684134 +0000 UTC m=+2046.760127133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls") pod "maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" (UID: "cb354199-e405-4f1f-a4ea-a95d238852bd") : secret "maas-api-e2e-ipp-a-a17de2-serving-cert" not found
Sep 16 13:12:24.177217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.177200    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb354199-e405-4f1f-a4ea-a95d238852bd-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.179086 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.179060    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-metrics-certs\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.186837 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.186813    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwtb4\" (UniqueName: \"kubernetes.io/projected/cb354199-e405-4f1f-a4ea-a95d238852bd-kube-api-access-pwtb4\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.680751 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.680712    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.683287 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.683266    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:24.938624 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:24.938537    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:25.210867 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:25.210747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"]
Sep 16 13:12:25.213525 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:12:25.213490    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb354199_e405_4f1f_a4ea_a95d238852bd.slice/crio-1d00e61369c40f5781880845472db08a3c4d77819484faa14af3f587a75c26a3 WatchSource:0}: Error finding container 1d00e61369c40f5781880845472db08a3c4d77819484faa14af3f587a75c26a3: Status 404 returned error can't find the container with id 1d00e61369c40f5781880845472db08a3c4d77819484faa14af3f587a75c26a3
Sep 16 13:12:25.214909 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:25.214890    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:12:25.255358 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:25.255307    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" event={"ID":"cb354199-e405-4f1f-a4ea-a95d238852bd","Type":"ContainerStarted","Data":"1d00e61369c40f5781880845472db08a3c4d77819484faa14af3f587a75c26a3"}
Sep 16 13:12:26.265210 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:26.265174    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" event={"ID":"cb354199-e405-4f1f-a4ea-a95d238852bd","Type":"ContainerStarted","Data":"a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea"}
Sep 16 13:12:26.265667 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:26.265276    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:26.282203 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:26.282156    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" podStartSLOduration=1.696564345 podStartE2EDuration="2.28214059s" totalImagesPullingTime="585.576245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:26.281210688 +0000 UTC m=+2048.364653732" watchObservedRunningTime="2026-09-16 13:12:26.28214059 +0000 UTC m=+2048.365583603"
Sep 16 13:12:32.275862 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:32.275832    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:12:32.700011 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:32.699948    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh" podUID="6cb12616-fa9a-465b-8de4-121412078cb4" containerName="main" containerID="cri-o://c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473" gracePeriod=45
Sep 16 13:12:32.946182 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:32.946151    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:12:33.064610 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064519    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-tmp-dir\") pod \"6cb12616-fa9a-465b-8de4-121412078cb4\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") "
Sep 16 13:12:33.064610 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064565    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb12616-fa9a-465b-8de4-121412078cb4-tls-certs\") pod \"6cb12616-fa9a-465b-8de4-121412078cb4\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") "
Sep 16 13:12:33.064847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064655    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-model-cache\") pod \"6cb12616-fa9a-465b-8de4-121412078cb4\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") "
Sep 16 13:12:33.064847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064742    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-home\") pod \"6cb12616-fa9a-465b-8de4-121412078cb4\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") "
Sep 16 13:12:33.064847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-dshm\") pod \"6cb12616-fa9a-465b-8de4-121412078cb4\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") "
Sep 16 13:12:33.064847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064801    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6cb12616-fa9a-465b-8de4-121412078cb4" (UID: "6cb12616-fa9a-465b-8de4-121412078cb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:33.064847 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064820    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6rbt\" (UniqueName: \"kubernetes.io/projected/6cb12616-fa9a-465b-8de4-121412078cb4-kube-api-access-b6rbt\") pod \"6cb12616-fa9a-465b-8de4-121412078cb4\" (UID: \"6cb12616-fa9a-465b-8de4-121412078cb4\") "
Sep 16 13:12:33.065115 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064883    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cb12616-fa9a-465b-8de4-121412078cb4" (UID: "6cb12616-fa9a-465b-8de4-121412078cb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:33.065115 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.064996    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-home" (OuterVolumeSpecName: "home") pod "6cb12616-fa9a-465b-8de4-121412078cb4" (UID: "6cb12616-fa9a-465b-8de4-121412078cb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:33.065215 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.065143    2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-model-cache\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:12:33.065215 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.065162    2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-home\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:12:33.065215 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.065171    2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-tmp-dir\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:12:33.067030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.067000    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb12616-fa9a-465b-8de4-121412078cb4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cb12616-fa9a-465b-8de4-121412078cb4" (UID: "6cb12616-fa9a-465b-8de4-121412078cb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:12:33.067030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.067004    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-dshm" (OuterVolumeSpecName: "dshm") pod "6cb12616-fa9a-465b-8de4-121412078cb4" (UID: "6cb12616-fa9a-465b-8de4-121412078cb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 13:12:33.067158 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.067102    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb12616-fa9a-465b-8de4-121412078cb4-kube-api-access-b6rbt" (OuterVolumeSpecName: "kube-api-access-b6rbt") pod "6cb12616-fa9a-465b-8de4-121412078cb4" (UID: "6cb12616-fa9a-465b-8de4-121412078cb4"). InnerVolumeSpecName "kube-api-access-b6rbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:12:33.165726 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.165689    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6rbt\" (UniqueName: \"kubernetes.io/projected/6cb12616-fa9a-465b-8de4-121412078cb4-kube-api-access-b6rbt\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:12:33.165726 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.165721    2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cb12616-fa9a-465b-8de4-121412078cb4-tls-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:12:33.165726 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.165733    2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cb12616-fa9a-465b-8de4-121412078cb4-dshm\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:12:33.302549 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.302514    2559 generic.go:356] "Generic (PLEG): container finished" podID="6cb12616-fa9a-465b-8de4-121412078cb4" containerID="c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473" exitCode=0
Sep 16 13:12:33.302949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.302558    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"
Sep 16 13:12:33.302949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.302589    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh" event={"ID":"6cb12616-fa9a-465b-8de4-121412078cb4","Type":"ContainerDied","Data":"c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473"}
Sep 16 13:12:33.302949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.302627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh" event={"ID":"6cb12616-fa9a-465b-8de4-121412078cb4","Type":"ContainerDied","Data":"87d7b3678314cde6652553f000fb7db876336207ad1a60eed9ac29c1efa02411"}
Sep 16 13:12:33.302949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.302644    2559 scope.go:117] "RemoveContainer" containerID="c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473"
Sep 16 13:12:33.312242 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.312225    2559 scope.go:117] "RemoveContainer" containerID="c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473"
Sep 16 13:12:33.312537 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:12:33.312514    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473\": container with ID starting with c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473 not found: ID does not exist" containerID="c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473"
Sep 16 13:12:33.312587 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.312546    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473"} err="failed to get container status \"c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473\": rpc error: code = NotFound desc = could not find container \"c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473\": container with ID starting with c558c5e788f0aee7b1dbe5ab1757ff41361e4ba9d517d09964b91ddf81890473 not found: ID does not exist"
Sep 16 13:12:33.328161 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.328106    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"]
Sep 16 13:12:33.331473 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:33.331450    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1ef95c/rate-test-model-1ef95c-kserve-7596487f79-77fxh"]
Sep 16 13:12:34.411739 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:34.411705    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb12616-fa9a-465b-8de4-121412078cb4" path="/var/lib/kubelet/pods/6cb12616-fa9a-465b-8de4-121412078cb4/volumes"
Sep 16 13:12:34.842427 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:34.842394    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:12:53.795027 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.794940    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"]
Sep 16 13:12:53.795596 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.795577    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb12616-fa9a-465b-8de4-121412078cb4" containerName="main"
Sep 16 13:12:53.795653 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.795600    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb12616-fa9a-465b-8de4-121412078cb4" containerName="main"
Sep 16 13:12:53.795733 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.795721    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb12616-fa9a-465b-8de4-121412078cb4" containerName="main"
Sep 16 13:12:53.799125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.799103    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.801101 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.801081    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8d2eac-serving-cert\""
Sep 16 13:12:53.809745 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.809717    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"]
Sep 16 13:12:53.865796 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.865762    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a102c71a-9848-455e-ab49-63f03c289719-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.865796 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.865802    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.866066 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.865830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.866066 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.865954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65n7\" (UniqueName: \"kubernetes.io/projected/a102c71a-9848-455e-ab49-63f03c289719-kube-api-access-c65n7\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.967093 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.967058    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65n7\" (UniqueName: \"kubernetes.io/projected/a102c71a-9848-455e-ab49-63f03c289719-kube-api-access-c65n7\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.967270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.967130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a102c71a-9848-455e-ab49-63f03c289719-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.967270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.967156    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.967270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.967188    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.967855 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.967827    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a102c71a-9848-455e-ab49-63f03c289719-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.969747 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.969726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-metrics-certs\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.969852 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.969755    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:53.975212 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:53.975191    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65n7\" (UniqueName: \"kubernetes.io/projected/a102c71a-9848-455e-ab49-63f03c289719-kube-api-access-c65n7\") pod \"maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:54.112214 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.112122    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:54.386777 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.386750    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"]
Sep 16 13:12:54.388817 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:12:54.388781    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda102c71a_9848_455e_ab49_63f03c289719.slice/crio-6b78e0346fe552cca080f98c00947b6e3d2aa712fac88c2c8374af35021e7669 WatchSource:0}: Error finding container 6b78e0346fe552cca080f98c00947b6e3d2aa712fac88c2c8374af35021e7669: Status 404 returned error can't find the container with id 6b78e0346fe552cca080f98c00947b6e3d2aa712fac88c2c8374af35021e7669
Sep 16 13:12:54.404533 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.404505    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq" event={"ID":"a102c71a-9848-455e-ab49-63f03c289719","Type":"ContainerStarted","Data":"6b78e0346fe552cca080f98c00947b6e3d2aa712fac88c2c8374af35021e7669"}
Sep 16 13:12:54.538274 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.538241    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"]
Sep 16 13:12:54.542823 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.542802    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.545206 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.545184    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8d2eac-dockercfg-p5qrw\""
Sep 16 13:12:54.545354 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.545257    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8d2eac\""
Sep 16 13:12:54.552786 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.552760    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"]
Sep 16 13:12:54.674517 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.674476    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1162330c-8fd0-4732-8ace-c78f71e94409-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.674683 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.674539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwqbr\" (UniqueName: \"kubernetes.io/projected/1162330c-8fd0-4732-8ace-c78f71e94409-kube-api-access-lwqbr\") pod \"payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.775191 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.775142    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwqbr\" (UniqueName: \"kubernetes.io/projected/1162330c-8fd0-4732-8ace-c78f71e94409-kube-api-access-lwqbr\") pod \"payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.775413 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.775282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1162330c-8fd0-4732-8ace-c78f71e94409-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.776144 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.776124    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1162330c-8fd0-4732-8ace-c78f71e94409-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.782721 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.782699    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwqbr\" (UniqueName: \"kubernetes.io/projected/1162330c-8fd0-4732-8ace-c78f71e94409-kube-api-access-lwqbr\") pod \"payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:54.855386 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:54.855347    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:55.128795 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.128714    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"]
Sep 16 13:12:55.132345 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:12:55.132292    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1162330c_8fd0_4732_8ace_c78f71e94409.slice/crio-ad2b26fdd8bba7ae0f64eb73238ee0e2851428a4ab1a465de950daa736cfa432 WatchSource:0}: Error finding container ad2b26fdd8bba7ae0f64eb73238ee0e2851428a4ab1a465de950daa736cfa432: Status 404 returned error can't find the container with id ad2b26fdd8bba7ae0f64eb73238ee0e2851428a4ab1a465de950daa736cfa432
Sep 16 13:12:55.410787 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.410692    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s" event={"ID":"1162330c-8fd0-4732-8ace-c78f71e94409","Type":"ContainerStarted","Data":"878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4"}
Sep 16 13:12:55.410952 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.410800    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:12:55.410952 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.410830    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s" event={"ID":"1162330c-8fd0-4732-8ace-c78f71e94409","Type":"ContainerStarted","Data":"ad2b26fdd8bba7ae0f64eb73238ee0e2851428a4ab1a465de950daa736cfa432"}
Sep 16 13:12:55.412373 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.412350    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq" event={"ID":"a102c71a-9848-455e-ab49-63f03c289719","Type":"ContainerStarted","Data":"eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136"}
Sep 16 13:12:55.412506 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.412473    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:12:55.427808 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.427757    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s" podStartSLOduration=1.4277423009999999 podStartE2EDuration="1.427742301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:55.426477157 +0000 UTC m=+2077.509920187" watchObservedRunningTime="2026-09-16 13:12:55.427742301 +0000 UTC m=+2077.511185315"
Sep 16 13:12:55.443494 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:55.443440    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq" podStartSLOduration=2.128424531 podStartE2EDuration="2.443424287s" totalImagesPullingTime="314.999756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:12:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:12:55.441346073 +0000 UTC m=+2077.524789100" watchObservedRunningTime="2026-09-16 13:12:55.443424287 +0000 UTC m=+2077.526867301"
Sep 16 13:12:59.350960 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:12:59.350922    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:13:01.422999 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:13:01.422966    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:13:06.419169 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:13:06.419140    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:13:26.846363 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:13:26.846309    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s_1162330c-8fd0-4732-8ace-c78f71e94409/payload-processing/0.log"
Sep 16 13:13:34.844788 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:13:34.844747    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:13:56.748564 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:13:56.748528    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:14:03.193064 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:14:03.193033    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s_1162330c-8fd0-4732-8ace-c78f71e94409/payload-processing/0.log"
Sep 16 13:14:03.315073 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:14:03.315046    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s_1162330c-8fd0-4732-8ace-c78f71e94409/payload-processing/0.log"
Sep 16 13:14:16.253200 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:14:16.253115    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:14:53.544662 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:14:53.544621    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:15:00.147473 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:00.147435    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"]
Sep 16 13:15:00.151177 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:00.151150    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"
Sep 16 13:15:00.153502 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:00.153478    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-64zph\""
Sep 16 13:15:00.162462 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:00.162436    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"]
Sep 16 13:15:00.165030 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:00.165010    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"
Sep 16 13:15:00.434456 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:00.434427    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"]
Sep 16 13:15:00.437850 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:15:00.437823    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f1c7cd_097d_4499_b04c_e81813218220.slice/crio-640ff5cf218960a7061dd62e1c4630cd368540799f949fdd5efe1ea7fbb9afbc WatchSource:0}: Error finding container 640ff5cf218960a7061dd62e1c4630cd368540799f949fdd5efe1ea7fbb9afbc: Status 404 returned error can't find the container with id 640ff5cf218960a7061dd62e1c4630cd368540799f949fdd5efe1ea7fbb9afbc
Sep 16 13:15:01.027098 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:01.027002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8" event={"ID":"b1f1c7cd-097d-4499-b04c-e81813218220","Type":"ContainerStarted","Data":"b991f51fc043a868d0ee6f9f332fb1a2639a0bb43b84366b5533e599e991bfc6"}
Sep 16 13:15:01.027098 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:01.027069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8" event={"ID":"b1f1c7cd-097d-4499-b04c-e81813218220","Type":"ContainerStarted","Data":"640ff5cf218960a7061dd62e1c4630cd368540799f949fdd5efe1ea7fbb9afbc"}
Sep 16 13:15:01.043873 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:01.043822    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8" podStartSLOduration=1.043804814 podStartE2EDuration="1.043804814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:15:01.04164311 +0000 UTC m=+2203.125086144" watchObservedRunningTime="2026-09-16 13:15:01.043804814 +0000 UTC m=+2203.127247884"
Sep 16 13:15:02.033605 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:02.033570    2559 generic.go:356] "Generic (PLEG): container finished" podID="b1f1c7cd-097d-4499-b04c-e81813218220" containerID="b991f51fc043a868d0ee6f9f332fb1a2639a0bb43b84366b5533e599e991bfc6" exitCode=0
Sep 16 13:15:02.034007 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:02.033619    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8" event={"ID":"b1f1c7cd-097d-4499-b04c-e81813218220","Type":"ContainerDied","Data":"b991f51fc043a868d0ee6f9f332fb1a2639a0bb43b84366b5533e599e991bfc6"}
Sep 16 13:15:03.173640 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:03.173603    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"
Sep 16 13:15:04.044537 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:04.044501    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8" event={"ID":"b1f1c7cd-097d-4499-b04c-e81813218220","Type":"ContainerDied","Data":"640ff5cf218960a7061dd62e1c4630cd368540799f949fdd5efe1ea7fbb9afbc"}
Sep 16 13:15:04.044537 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:04.044525    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"
Sep 16 13:15:04.044763 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:04.044551    2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640ff5cf218960a7061dd62e1c4630cd368540799f949fdd5efe1ea7fbb9afbc"
Sep 16 13:15:12.754078 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:12.754043    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:15:15.181887 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.181855    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"]
Sep 16 13:15:15.182302 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.182118    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" podUID="cb354199-e405-4f1f-a4ea-a95d238852bd" containerName="maas-api" containerID="cri-o://a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea" gracePeriod=30
Sep 16 13:15:15.219120 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.219092    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"]
Sep 16 13:15:15.222023 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.221997    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a17de2-29826075-tgxr8"]
Sep 16 13:15:15.439251 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.439192    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:15:15.531700 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.531660    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls\") pod \"cb354199-e405-4f1f-a4ea-a95d238852bd\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") "
Sep 16 13:15:15.531897 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.531761    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb354199-e405-4f1f-a4ea-a95d238852bd-openshift-service-ca\") pod \"cb354199-e405-4f1f-a4ea-a95d238852bd\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") "
Sep 16 13:15:15.531897 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.531879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwtb4\" (UniqueName: \"kubernetes.io/projected/cb354199-e405-4f1f-a4ea-a95d238852bd-kube-api-access-pwtb4\") pod \"cb354199-e405-4f1f-a4ea-a95d238852bd\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") "
Sep 16 13:15:15.532031 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.531922    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-metrics-certs\") pod \"cb354199-e405-4f1f-a4ea-a95d238852bd\" (UID: \"cb354199-e405-4f1f-a4ea-a95d238852bd\") "
Sep 16 13:15:15.532168 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.532141    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb354199-e405-4f1f-a4ea-a95d238852bd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb354199-e405-4f1f-a4ea-a95d238852bd" (UID: "cb354199-e405-4f1f-a4ea-a95d238852bd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:15:15.532255 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.532236    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb354199-e405-4f1f-a4ea-a95d238852bd-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.534129 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.534100    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb354199-e405-4f1f-a4ea-a95d238852bd" (UID: "cb354199-e405-4f1f-a4ea-a95d238852bd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:15.534310 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.534288    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb354199-e405-4f1f-a4ea-a95d238852bd-kube-api-access-pwtb4" (OuterVolumeSpecName: "kube-api-access-pwtb4") pod "cb354199-e405-4f1f-a4ea-a95d238852bd" (UID: "cb354199-e405-4f1f-a4ea-a95d238852bd"). InnerVolumeSpecName "kube-api-access-pwtb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:15:15.534436 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.534350    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb354199-e405-4f1f-a4ea-a95d238852bd" (UID: "cb354199-e405-4f1f-a4ea-a95d238852bd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:15:15.633126 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.633088    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.633126 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.633120    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwtb4\" (UniqueName: \"kubernetes.io/projected/cb354199-e405-4f1f-a4ea-a95d238852bd-kube-api-access-pwtb4\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:15:15.633126 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:15.633131    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb354199-e405-4f1f-a4ea-a95d238852bd-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:15:16.106823 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.106785    2559 generic.go:356] "Generic (PLEG): container finished" podID="cb354199-e405-4f1f-a4ea-a95d238852bd" containerID="a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea" exitCode=0
Sep 16 13:15:16.107025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.106846    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"
Sep 16 13:15:16.107025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.106869    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" event={"ID":"cb354199-e405-4f1f-a4ea-a95d238852bd","Type":"ContainerDied","Data":"a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea"}
Sep 16 13:15:16.107025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.106952    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5" event={"ID":"cb354199-e405-4f1f-a4ea-a95d238852bd","Type":"ContainerDied","Data":"1d00e61369c40f5781880845472db08a3c4d77819484faa14af3f587a75c26a3"}
Sep 16 13:15:16.107025 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.106975    2559 scope.go:117] "RemoveContainer" containerID="a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea"
Sep 16 13:15:16.116258 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.116238    2559 scope.go:117] "RemoveContainer" containerID="a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea"
Sep 16 13:15:16.116560 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:15:16.116541    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea\": container with ID starting with a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea not found: ID does not exist" containerID="a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea"
Sep 16 13:15:16.116623 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.116569    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea"} err="failed to get container status \"a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea\": rpc error: code = NotFound desc = could not find container \"a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea\": container with ID starting with a78b17101f9c3a2f2a14c5388d35756e9be874fd02f6b5bab0790e91a082dbea not found: ID does not exist"
Sep 16 13:15:16.126690 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.126667    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"]
Sep 16 13:15:16.132143 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.132123    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a17de2-7f66b65bd5-s59k5"]
Sep 16 13:15:16.412160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.412133    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f1c7cd-097d-4499-b04c-e81813218220" path="/var/lib/kubelet/pods/b1f1c7cd-097d-4499-b04c-e81813218220/volumes"
Sep 16 13:15:16.412550 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:16.412536    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb354199-e405-4f1f-a4ea-a95d238852bd" path="/var/lib/kubelet/pods/cb354199-e405-4f1f-a4ea-a95d238852bd/volumes"
Sep 16 13:15:25.557181 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:15:25.557138    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:16:15.992945 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:15.992911    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"]
Sep 16 13:16:15.993474 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:15.993172    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq" podUID="a102c71a-9848-455e-ab49-63f03c289719" containerName="maas-api" containerID="cri-o://eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136" gracePeriod=30
Sep 16 13:16:16.075834 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.075799    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"]
Sep 16 13:16:16.076239 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.076094    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s" podUID="1162330c-8fd0-4732-8ace-c78f71e94409" containerName="payload-processing" containerID="cri-o://878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4" gracePeriod=30
Sep 16 13:16:16.344290 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.344266    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:16:16.347054 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.347032    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:16:16.412966 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.412929    2559 generic.go:356] "Generic (PLEG): container finished" podID="a102c71a-9848-455e-ab49-63f03c289719" containerID="eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136" exitCode=0
Sep 16 13:16:16.413164 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.413044    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"
Sep 16 13:16:16.414485 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.414463    2559 generic.go:356] "Generic (PLEG): container finished" podID="1162330c-8fd0-4732-8ace-c78f71e94409" containerID="878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4" exitCode=0
Sep 16 13:16:16.414595 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.414551    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"
Sep 16 13:16:16.415754 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.415738    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq" event={"ID":"a102c71a-9848-455e-ab49-63f03c289719","Type":"ContainerDied","Data":"eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136"}
Sep 16 13:16:16.415799 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.415784    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq" event={"ID":"a102c71a-9848-455e-ab49-63f03c289719","Type":"ContainerDied","Data":"6b78e0346fe552cca080f98c00947b6e3d2aa712fac88c2c8374af35021e7669"}
Sep 16 13:16:16.415831 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.415806    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s" event={"ID":"1162330c-8fd0-4732-8ace-c78f71e94409","Type":"ContainerDied","Data":"878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4"}
Sep 16 13:16:16.415831 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.415823    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s" event={"ID":"1162330c-8fd0-4732-8ace-c78f71e94409","Type":"ContainerDied","Data":"ad2b26fdd8bba7ae0f64eb73238ee0e2851428a4ab1a465de950daa736cfa432"}
Sep 16 13:16:16.415895 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.415843    2559 scope.go:117] "RemoveContainer" containerID="eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136"
Sep 16 13:16:16.424980 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.424966    2559 scope.go:117] "RemoveContainer" containerID="eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136"
Sep 16 13:16:16.425226 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:16:16.425208    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136\": container with ID starting with eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136 not found: ID does not exist" containerID="eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136"
Sep 16 13:16:16.425266 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.425235    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136"} err="failed to get container status \"eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136\": rpc error: code = NotFound desc = could not find container \"eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136\": container with ID starting with eeb6bafdca14e893b9cb816fd619f7d263101a489601ab80e1da32e2f0299136 not found: ID does not exist"
Sep 16 13:16:16.425266 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.425250    2559 scope.go:117] "RemoveContainer" containerID="878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4"
Sep 16 13:16:16.432760 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.432745    2559 scope.go:117] "RemoveContainer" containerID="878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4"
Sep 16 13:16:16.433024 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:16:16.433006    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4\": container with ID starting with 878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4 not found: ID does not exist" containerID="878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4"
Sep 16 13:16:16.433075 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.433029    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4"} err="failed to get container status \"878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4\": rpc error: code = NotFound desc = could not find container \"878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4\": container with ID starting with 878253d3a7f95610a8db2ad00156d02ed8cb99bd691fee503bee8f1c02de1dd4 not found: ID does not exist"
Sep 16 13:16:16.503560 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.503529    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c65n7\" (UniqueName: \"kubernetes.io/projected/a102c71a-9848-455e-ab49-63f03c289719-kube-api-access-c65n7\") pod \"a102c71a-9848-455e-ab49-63f03c289719\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") "
Sep 16 13:16:16.503748 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.503593    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1162330c-8fd0-4732-8ace-c78f71e94409-plugins-config-volume\") pod \"1162330c-8fd0-4732-8ace-c78f71e94409\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") "
Sep 16 13:16:16.503748 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.503621    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwqbr\" (UniqueName: \"kubernetes.io/projected/1162330c-8fd0-4732-8ace-c78f71e94409-kube-api-access-lwqbr\") pod \"1162330c-8fd0-4732-8ace-c78f71e94409\" (UID: \"1162330c-8fd0-4732-8ace-c78f71e94409\") "
Sep 16 13:16:16.503748 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.503663    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-metrics-certs\") pod \"a102c71a-9848-455e-ab49-63f03c289719\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") "
Sep 16 13:16:16.503748 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.503704    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a102c71a-9848-455e-ab49-63f03c289719-openshift-service-ca\") pod \"a102c71a-9848-455e-ab49-63f03c289719\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") "
Sep 16 13:16:16.503748 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.503729    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-maas-api-tls\") pod \"a102c71a-9848-455e-ab49-63f03c289719\" (UID: \"a102c71a-9848-455e-ab49-63f03c289719\") "
Sep 16 13:16:16.504094 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.504024    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1162330c-8fd0-4732-8ace-c78f71e94409-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1162330c-8fd0-4732-8ace-c78f71e94409" (UID: "1162330c-8fd0-4732-8ace-c78f71e94409"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:16.504160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.504105    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a102c71a-9848-455e-ab49-63f03c289719-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a102c71a-9848-455e-ab49-63f03c289719" (UID: "a102c71a-9848-455e-ab49-63f03c289719"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:16.506023 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.505992    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a102c71a-9848-455e-ab49-63f03c289719-kube-api-access-c65n7" (OuterVolumeSpecName: "kube-api-access-c65n7") pod "a102c71a-9848-455e-ab49-63f03c289719" (UID: "a102c71a-9848-455e-ab49-63f03c289719"). InnerVolumeSpecName "kube-api-access-c65n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:16.506136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.506026    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a102c71a-9848-455e-ab49-63f03c289719" (UID: "a102c71a-9848-455e-ab49-63f03c289719"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:16:16.506136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.506050    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1162330c-8fd0-4732-8ace-c78f71e94409-kube-api-access-lwqbr" (OuterVolumeSpecName: "kube-api-access-lwqbr") pod "1162330c-8fd0-4732-8ace-c78f71e94409" (UID: "1162330c-8fd0-4732-8ace-c78f71e94409"). InnerVolumeSpecName "kube-api-access-lwqbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:16.506136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.506052    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a102c71a-9848-455e-ab49-63f03c289719" (UID: "a102c71a-9848-455e-ab49-63f03c289719"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:16:16.604813 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.604731    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.604813 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.604763    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65n7\" (UniqueName: \"kubernetes.io/projected/a102c71a-9848-455e-ab49-63f03c289719-kube-api-access-c65n7\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.604813 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.604772    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1162330c-8fd0-4732-8ace-c78f71e94409-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.604813 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.604781    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwqbr\" (UniqueName: \"kubernetes.io/projected/1162330c-8fd0-4732-8ace-c78f71e94409-kube-api-access-lwqbr\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.604813 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.604791    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a102c71a-9848-455e-ab49-63f03c289719-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.604813 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.604800    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a102c71a-9848-455e-ab49-63f03c289719-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:16.740350 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.740305    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"]
Sep 16 13:16:16.744415 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.744383    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8d2eac-795f99bdd9-2kckq"]
Sep 16 13:16:16.753428 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.753401    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"]
Sep 16 13:16:16.759335 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:16.759286    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8d2eac-7c8c9987cc-d627s"]
Sep 16 13:16:18.413148 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:18.413111    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1162330c-8fd0-4732-8ace-c78f71e94409" path="/var/lib/kubelet/pods/1162330c-8fd0-4732-8ace-c78f71e94409/volumes"
Sep 16 13:16:18.413529 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:18.413519    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a102c71a-9848-455e-ab49-63f03c289719" path="/var/lib/kubelet/pods/a102c71a-9848-455e-ab49-63f03c289719/volumes"
Sep 16 13:16:19.277510 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:19.277474    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:16:37.659995 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:37.659952    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:16:52.913598 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:52.913565    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"]
Sep 16 13:16:52.914098 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:52.913761    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" podUID="1aa3918b-ea6d-4352-8d49-74f1d524d007" containerName="maas-api" containerID="cri-o://59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e" gracePeriod=30
Sep 16 13:16:53.017967 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.017932    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"]
Sep 16 13:16:53.018214 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.018186    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb" podUID="ee86e007-bb68-4d4d-8419-fdd892c34d9c" containerName="payload-pre-processing" containerID="cri-o://a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726" gracePeriod=30
Sep 16 13:16:53.172089 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.172023    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:16:53.249276 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.249244    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-metrics-certs\") pod \"1aa3918b-ea6d-4352-8d49-74f1d524d007\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") "
Sep 16 13:16:53.249276 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.249279    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls\") pod \"1aa3918b-ea6d-4352-8d49-74f1d524d007\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") "
Sep 16 13:16:53.249538 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.249300    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3918b-ea6d-4352-8d49-74f1d524d007-openshift-service-ca\") pod \"1aa3918b-ea6d-4352-8d49-74f1d524d007\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") "
Sep 16 13:16:53.249538 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.249377    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5nkn\" (UniqueName: \"kubernetes.io/projected/1aa3918b-ea6d-4352-8d49-74f1d524d007-kube-api-access-b5nkn\") pod \"1aa3918b-ea6d-4352-8d49-74f1d524d007\" (UID: \"1aa3918b-ea6d-4352-8d49-74f1d524d007\") "
Sep 16 13:16:53.249744 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.249712    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa3918b-ea6d-4352-8d49-74f1d524d007-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1aa3918b-ea6d-4352-8d49-74f1d524d007" (UID: "1aa3918b-ea6d-4352-8d49-74f1d524d007"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:53.251709 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.251680    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1aa3918b-ea6d-4352-8d49-74f1d524d007" (UID: "1aa3918b-ea6d-4352-8d49-74f1d524d007"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:16:53.251810 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.251732    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1aa3918b-ea6d-4352-8d49-74f1d524d007" (UID: "1aa3918b-ea6d-4352-8d49-74f1d524d007"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:16:53.251810 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.251745    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa3918b-ea6d-4352-8d49-74f1d524d007-kube-api-access-b5nkn" (OuterVolumeSpecName: "kube-api-access-b5nkn") pod "1aa3918b-ea6d-4352-8d49-74f1d524d007" (UID: "1aa3918b-ea6d-4352-8d49-74f1d524d007"). InnerVolumeSpecName "kube-api-access-b5nkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:53.283038 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.283015    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:16:53.349908 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.349881    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee86e007-bb68-4d4d-8419-fdd892c34d9c-plugins-config-volume\") pod \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") "
Sep 16 13:16:53.350078 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.350000    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwhff\" (UniqueName: \"kubernetes.io/projected/ee86e007-bb68-4d4d-8419-fdd892c34d9c-kube-api-access-cwhff\") pod \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\" (UID: \"ee86e007-bb68-4d4d-8419-fdd892c34d9c\") "
Sep 16 13:16:53.350193 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.350179    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.350232 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.350198    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aa3918b-ea6d-4352-8d49-74f1d524d007-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.350232 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.350208    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aa3918b-ea6d-4352-8d49-74f1d524d007-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.350232 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.350217    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5nkn\" (UniqueName: \"kubernetes.io/projected/1aa3918b-ea6d-4352-8d49-74f1d524d007-kube-api-access-b5nkn\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.350352 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.350285    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee86e007-bb68-4d4d-8419-fdd892c34d9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee86e007-bb68-4d4d-8419-fdd892c34d9c" (UID: "ee86e007-bb68-4d4d-8419-fdd892c34d9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:16:53.352181 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.352164    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee86e007-bb68-4d4d-8419-fdd892c34d9c-kube-api-access-cwhff" (OuterVolumeSpecName: "kube-api-access-cwhff") pod "ee86e007-bb68-4d4d-8419-fdd892c34d9c" (UID: "ee86e007-bb68-4d4d-8419-fdd892c34d9c"). InnerVolumeSpecName "kube-api-access-cwhff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:16:53.451638 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.451558    2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee86e007-bb68-4d4d-8419-fdd892c34d9c-plugins-config-volume\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.451638 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.451582    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhff\" (UniqueName: \"kubernetes.io/projected/ee86e007-bb68-4d4d-8419-fdd892c34d9c-kube-api-access-cwhff\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:16:53.597705 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.597671    2559 generic.go:356] "Generic (PLEG): container finished" podID="ee86e007-bb68-4d4d-8419-fdd892c34d9c" containerID="a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726" exitCode=0
Sep 16 13:16:53.597888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.597727    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"
Sep 16 13:16:53.597888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.597755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb" event={"ID":"ee86e007-bb68-4d4d-8419-fdd892c34d9c","Type":"ContainerDied","Data":"a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726"}
Sep 16 13:16:53.597888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.597793    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb" event={"ID":"ee86e007-bb68-4d4d-8419-fdd892c34d9c","Type":"ContainerDied","Data":"5406de4744a0d53f11642815eca05ed5fe9f774bb0b8ffac6038f0a962ae6dbc"}
Sep 16 13:16:53.597888 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.597820    2559 scope.go:117] "RemoveContainer" containerID="a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726"
Sep 16 13:16:53.599149 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.599129    2559 generic.go:356] "Generic (PLEG): container finished" podID="1aa3918b-ea6d-4352-8d49-74f1d524d007" containerID="59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e" exitCode=0
Sep 16 13:16:53.599228 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.599166    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"
Sep 16 13:16:53.599228 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.599177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" event={"ID":"1aa3918b-ea6d-4352-8d49-74f1d524d007","Type":"ContainerDied","Data":"59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e"}
Sep 16 13:16:53.599228 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.599212    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd" event={"ID":"1aa3918b-ea6d-4352-8d49-74f1d524d007","Type":"ContainerDied","Data":"4d4db086fcd68c3e3781b549ad8d70d72cc3375f96ff26a60fe4b859bf88c914"}
Sep 16 13:16:53.607989 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.607974    2559 scope.go:117] "RemoveContainer" containerID="a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726"
Sep 16 13:16:53.608254 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:16:53.608231    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726\": container with ID starting with a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726 not found: ID does not exist" containerID="a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726"
Sep 16 13:16:53.608374 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.608265    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726"} err="failed to get container status \"a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726\": rpc error: code = NotFound desc = could not find container \"a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726\": container with ID starting with a6e36120c764f7cc59edd2f6253ca6bd54ab4f607716c3d3aaf2839f6706f726 not found: ID does not exist"
Sep 16 13:16:53.608374 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.608286    2559 scope.go:117] "RemoveContainer" containerID="59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e"
Sep 16 13:16:53.616479 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.616461    2559 scope.go:117] "RemoveContainer" containerID="59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e"
Sep 16 13:16:53.616736 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:16:53.616720    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e\": container with ID starting with 59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e not found: ID does not exist" containerID="59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e"
Sep 16 13:16:53.616806 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.616740    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e"} err="failed to get container status \"59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e\": rpc error: code = NotFound desc = could not find container \"59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e\": container with ID starting with 59b3627baaed9f4894ea17969f240ab1b59eaad2706693e900e5b474aea4541e not found: ID does not exist"
Sep 16 13:16:53.655149 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.655122    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"]
Sep 16 13:16:53.665366 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.665342    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e344f9-fd4fdfd46-gcmqd"]
Sep 16 13:16:53.684972 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.684948    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"]
Sep 16 13:16:53.708056 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:53.708000    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e344f9-59f8ccd6f8-v9jzb"]
Sep 16 13:16:54.411772 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:54.411742    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aa3918b-ea6d-4352-8d49-74f1d524d007" path="/var/lib/kubelet/pods/1aa3918b-ea6d-4352-8d49-74f1d524d007/volumes"
Sep 16 13:16:54.412136 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:54.412084    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee86e007-bb68-4d4d-8419-fdd892c34d9c" path="/var/lib/kubelet/pods/ee86e007-bb68-4d4d-8419-fdd892c34d9c/volumes"
Sep 16 13:16:55.760823 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:16:55.760787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:17:04.790612 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:17:04.790578    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:18:25.759979 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:18:25.759942    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:18:50.742099 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:18:50.742014    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:18:58.659928 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:18:58.659891    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:19:05.765862 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:19:05.765831    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:19:12.944434 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:19:12.944398    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:19:27.742229 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:19:27.742192    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:19:37.956802 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:19:37.956759    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:21:05.161501 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:21:05.161462    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:21:13.155596 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:21:13.155561    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:21:18.839767 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:21:18.839729    2559 scope.go:117] "RemoveContainer" containerID="b991f51fc043a868d0ee6f9f332fb1a2639a0bb43b84366b5533e599e991bfc6"
Sep 16 13:21:19.357439 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:21:19.357403    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:21:38.951656 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:21:38.951623    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:21:55.155743 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:21:55.155703    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:22:39.154632 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:39.154597    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:22:48.652289 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:48.652252    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:22:55.449501 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.449466    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:22:55.948777 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.948747    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"]
Sep 16 13:22:55.949164 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949151    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a102c71a-9848-455e-ab49-63f03c289719" containerName="maas-api"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949168    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a102c71a-9848-455e-ab49-63f03c289719" containerName="maas-api"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949187    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee86e007-bb68-4d4d-8419-fdd892c34d9c" containerName="payload-pre-processing"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949192    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee86e007-bb68-4d4d-8419-fdd892c34d9c" containerName="payload-pre-processing"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949200    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aa3918b-ea6d-4352-8d49-74f1d524d007" containerName="maas-api"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949205    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aa3918b-ea6d-4352-8d49-74f1d524d007" containerName="maas-api"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949212    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1162330c-8fd0-4732-8ace-c78f71e94409" containerName="payload-processing"
Sep 16 13:22:55.949217 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949217    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1162330c-8fd0-4732-8ace-c78f71e94409" containerName="payload-processing"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949226    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb354199-e405-4f1f-a4ea-a95d238852bd" containerName="maas-api"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949231    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb354199-e405-4f1f-a4ea-a95d238852bd" containerName="maas-api"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949238    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f1c7cd-097d-4499-b04c-e81813218220" containerName="cleanup"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949243    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f1c7cd-097d-4499-b04c-e81813218220" containerName="cleanup"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949306    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aa3918b-ea6d-4352-8d49-74f1d524d007" containerName="maas-api"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949314    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee86e007-bb68-4d4d-8419-fdd892c34d9c" containerName="payload-pre-processing"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949337    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb354199-e405-4f1f-a4ea-a95d238852bd" containerName="maas-api"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949348    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a102c71a-9848-455e-ab49-63f03c289719" containerName="maas-api"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949354    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1162330c-8fd0-4732-8ace-c78f71e94409" containerName="payload-processing"
Sep 16 13:22:55.949452 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.949359    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f1c7cd-097d-4499-b04c-e81813218220" containerName="cleanup"
Sep 16 13:22:55.952360 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.952342    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:55.956538 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.956515    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 13:22:55.956674 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.956593    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 13:22:55.956674 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.956602    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nm49v\""
Sep 16 13:22:55.956674 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.956667    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 13:22:55.956843 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.956831    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 13:22:55.971000 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:55.970970    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"]
Sep 16 13:22:56.047558 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.047514    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-maas-api-tls\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.047558 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.047569    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-metrics-certs\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.047783 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.047694    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452a6a0e-2afc-41bd-a708-d35f6fc82f07-openshift-service-ca\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.047783 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.047745    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbp2\" (UniqueName: \"kubernetes.io/projected/452a6a0e-2afc-41bd-a708-d35f6fc82f07-kube-api-access-5pbp2\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.148901 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.148860    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-maas-api-tls\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.149064 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.148909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-metrics-certs\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.149064 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.148966    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452a6a0e-2afc-41bd-a708-d35f6fc82f07-openshift-service-ca\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.149064 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.149000    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbp2\" (UniqueName: \"kubernetes.io/projected/452a6a0e-2afc-41bd-a708-d35f6fc82f07-kube-api-access-5pbp2\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.149776 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.149685    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452a6a0e-2afc-41bd-a708-d35f6fc82f07-openshift-service-ca\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.151595 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.151567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-metrics-certs\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.151689 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.151597    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-maas-api-tls\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.156794 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.156774    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbp2\" (UniqueName: \"kubernetes.io/projected/452a6a0e-2afc-41bd-a708-d35f6fc82f07-kube-api-access-5pbp2\") pod \"maas-api-6d477c8649-8mwn6\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") " pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.263271 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.263181    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:56.539878 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.539806    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"]
Sep 16 13:22:56.542244 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:22:56.542209    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452a6a0e_2afc_41bd_a708_d35f6fc82f07.slice/crio-b064e0c1d37abf3c016184e437d14bc36f08971a1ff3cb1da1509e3fb72e628b WatchSource:0}: Error finding container b064e0c1d37abf3c016184e437d14bc36f08971a1ff3cb1da1509e3fb72e628b: Status 404 returned error can't find the container with id b064e0c1d37abf3c016184e437d14bc36f08971a1ff3cb1da1509e3fb72e628b
Sep 16 13:22:56.543617 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:56.543595    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 13:22:57.311184 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:57.311141    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6" event={"ID":"452a6a0e-2afc-41bd-a708-d35f6fc82f07","Type":"ContainerStarted","Data":"4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8"}
Sep 16 13:22:57.311388 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:57.311206    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6" event={"ID":"452a6a0e-2afc-41bd-a708-d35f6fc82f07","Type":"ContainerStarted","Data":"b064e0c1d37abf3c016184e437d14bc36f08971a1ff3cb1da1509e3fb72e628b"}
Sep 16 13:22:57.311388 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:57.311234    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:22:57.330715 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:22:57.330651    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6" podStartSLOduration=1.756295819 podStartE2EDuration="2.330633023s" totalImagesPullingTime="574.337204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:22:57.329669041 +0000 UTC m=+2679.413112084" watchObservedRunningTime="2026-09-16 13:22:57.330633023 +0000 UTC m=+2679.414076039"
Sep 16 13:23:03.320938 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:03.320907    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:23:03.358722 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:03.358682    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:23:04.208593 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.208559    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"]
Sep 16 13:23:04.212163 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.212135    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.227796 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.227755    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"]
Sep 16 13:23:04.325474 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.325438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-maas-api-tls\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.325474 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.325473    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-metrics-certs\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.325904 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.325492    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdz4r\" (UniqueName: \"kubernetes.io/projected/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-kube-api-access-jdz4r\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.325904 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.325532    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-openshift-service-ca\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.426666 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.426632    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-maas-api-tls\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.426666 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.426667    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-metrics-certs\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.426883 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.426684    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdz4r\" (UniqueName: \"kubernetes.io/projected/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-kube-api-access-jdz4r\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.426883 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.426709    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-openshift-service-ca\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.427457 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.427432    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-openshift-service-ca\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.429360 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.429306    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-maas-api-tls\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.429485 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.429407    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-metrics-certs\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.435131 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.435109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdz4r\" (UniqueName: \"kubernetes.io/projected/4361cc32-2b89-47b1-af4a-0cbd35ccfdc9-kube-api-access-jdz4r\") pod \"maas-api-684fb4cd97-grdq8\" (UID: \"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9\") " pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.522570 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.522467    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:04.794797 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:04.794768    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"]
Sep 16 13:23:04.797191 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:23:04.797161    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4361cc32_2b89_47b1_af4a_0cbd35ccfdc9.slice/crio-69d46f970e5d3c3ceee60c62d90b609fa5fd5d8ff0d706189d4e472a0bb84ab4 WatchSource:0}: Error finding container 69d46f970e5d3c3ceee60c62d90b609fa5fd5d8ff0d706189d4e472a0bb84ab4: Status 404 returned error can't find the container with id 69d46f970e5d3c3ceee60c62d90b609fa5fd5d8ff0d706189d4e472a0bb84ab4
Sep 16 13:23:05.349097 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:05.349005    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8" event={"ID":"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9","Type":"ContainerStarted","Data":"49e74646348ff4bf0dce3c0b5719349bf69d1ab3326278a727f2a7d9112e18d7"}
Sep 16 13:23:05.349097 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:05.349068    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8" event={"ID":"4361cc32-2b89-47b1-af4a-0cbd35ccfdc9","Type":"ContainerStarted","Data":"69d46f970e5d3c3ceee60c62d90b609fa5fd5d8ff0d706189d4e472a0bb84ab4"}
Sep 16 13:23:05.349515 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:05.349098    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:05.368001 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:05.367952    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8" podStartSLOduration=1.081742603 podStartE2EDuration="1.367934602s" totalImagesPullingTime="286.191999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 13:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:05.366573057 +0000 UTC m=+2687.450016090" watchObservedRunningTime="2026-09-16 13:23:05.367934602 +0000 UTC m=+2687.451377615"
Sep 16 13:23:10.549312 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:10.549232    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:23:11.359590 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.359559    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-684fb4cd97-grdq8"
Sep 16 13:23:11.410698 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.410662    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"]
Sep 16 13:23:11.410894 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.410850    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6" podUID="452a6a0e-2afc-41bd-a708-d35f6fc82f07" containerName="maas-api" containerID="cri-o://4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8" gracePeriod=30
Sep 16 13:23:11.652690 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.652664    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:23:11.694755 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.694719    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452a6a0e-2afc-41bd-a708-d35f6fc82f07-openshift-service-ca\") pod \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") "
Sep 16 13:23:11.694949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.694800    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-metrics-certs\") pod \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") "
Sep 16 13:23:11.694949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.694857    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pbp2\" (UniqueName: \"kubernetes.io/projected/452a6a0e-2afc-41bd-a708-d35f6fc82f07-kube-api-access-5pbp2\") pod \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") "
Sep 16 13:23:11.694949 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.694883    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-maas-api-tls\") pod \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\" (UID: \"452a6a0e-2afc-41bd-a708-d35f6fc82f07\") "
Sep 16 13:23:11.695123 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.695086    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452a6a0e-2afc-41bd-a708-d35f6fc82f07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "452a6a0e-2afc-41bd-a708-d35f6fc82f07" (UID: "452a6a0e-2afc-41bd-a708-d35f6fc82f07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 13:23:11.697150 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.697114    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "452a6a0e-2afc-41bd-a708-d35f6fc82f07" (UID: "452a6a0e-2afc-41bd-a708-d35f6fc82f07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:11.697270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.697190    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452a6a0e-2afc-41bd-a708-d35f6fc82f07-kube-api-access-5pbp2" (OuterVolumeSpecName: "kube-api-access-5pbp2") pod "452a6a0e-2afc-41bd-a708-d35f6fc82f07" (UID: "452a6a0e-2afc-41bd-a708-d35f6fc82f07"). InnerVolumeSpecName "kube-api-access-5pbp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 13:23:11.697270 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.697203    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "452a6a0e-2afc-41bd-a708-d35f6fc82f07" (UID: "452a6a0e-2afc-41bd-a708-d35f6fc82f07"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 13:23:11.796093 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.796062    2559 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-metrics-certs\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:23:11.796093 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.796092    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbp2\" (UniqueName: \"kubernetes.io/projected/452a6a0e-2afc-41bd-a708-d35f6fc82f07-kube-api-access-5pbp2\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:23:11.796093 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.796107    2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/452a6a0e-2afc-41bd-a708-d35f6fc82f07-maas-api-tls\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:23:11.796357 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:11.796119    2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/452a6a0e-2afc-41bd-a708-d35f6fc82f07-openshift-service-ca\") on node \"ip-10-0-143-72.ec2.internal\" DevicePath \"\""
Sep 16 13:23:12.386236 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.386204    2559 generic.go:356] "Generic (PLEG): container finished" podID="452a6a0e-2afc-41bd-a708-d35f6fc82f07" containerID="4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8" exitCode=0
Sep 16 13:23:12.386435 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.386250    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"
Sep 16 13:23:12.386435 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.386257    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6" event={"ID":"452a6a0e-2afc-41bd-a708-d35f6fc82f07","Type":"ContainerDied","Data":"4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8"}
Sep 16 13:23:12.386435 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.386310    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6" event={"ID":"452a6a0e-2afc-41bd-a708-d35f6fc82f07","Type":"ContainerDied","Data":"b064e0c1d37abf3c016184e437d14bc36f08971a1ff3cb1da1509e3fb72e628b"}
Sep 16 13:23:12.386435 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.386361    2559 scope.go:117] "RemoveContainer" containerID="4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8"
Sep 16 13:23:12.396093 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.396072    2559 scope.go:117] "RemoveContainer" containerID="4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8"
Sep 16 13:23:12.396386 ip-10-0-143-72 kubenswrapper[2559]: E0916 13:23:12.396366    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8\": container with ID starting with 4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8 not found: ID does not exist" containerID="4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8"
Sep 16 13:23:12.396442 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.396396    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8"} err="failed to get container status \"4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8\": rpc error: code = NotFound desc = could not find container \"4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8\": container with ID starting with 4df8b231797687e3610354d957338c07e3ed6843b891aa46c4bcc83b3df640e8 not found: ID does not exist"
Sep 16 13:23:12.411430 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.411405    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"]
Sep 16 13:23:12.413532 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:12.413506    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d477c8649-8mwn6"]
Sep 16 13:23:14.411629 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:14.411594    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452a6a0e-2afc-41bd-a708-d35f6fc82f07" path="/var/lib/kubelet/pods/452a6a0e-2afc-41bd-a708-d35f6fc82f07/volumes"
Sep 16 13:23:18.658492 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:18.658456    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:23:23.754553 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:23.754511    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-wxzvf"]
Sep 16 13:23:26.937764 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:26.937728    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb5d9656f-dxhbx_deb60e74-887e-46af-bc89-3216fd090dc7/authorino/0.log"
Sep 16 13:23:33.417668 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:33.417637    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-k7b8l_e746e6c0-4127-42e7-b184-d8e90274bf4d/copy-manifests/0.log"
Sep 16 13:23:33.424455 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:33.424432    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-k7b8l_e746e6c0-4127-42e7-b184-d8e90274bf4d/manager/0.log"
Sep 16 13:23:33.884625 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:33.884529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-684fb4cd97-grdq8_4361cc32-2b89-47b1-af4a-0cbd35ccfdc9/maas-api/0.log"
Sep 16 13:23:33.993041 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:33.993017    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29826060-2ckx2_f1131c3d-0451-43c2-8948-c7da878ab80a/cleanup/0.log"
Sep 16 13:23:35.075068 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:35.075037    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb5d9656f-dxhbx_deb60e74-887e-46af-bc89-3216fd090dc7/authorino/0.log"
Sep 16 13:23:35.483578 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:35.483548    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8vkn5_7bd517a2-175f-4f07-a62b-d5733ceca442/kuadrant-console-plugin/0.log"
Sep 16 13:23:35.735125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:35.735047    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wxzvf_7842f60e-a30c-497b-b11d-6841236c8dc0/limitador/0.log"
Sep 16 13:23:36.464920 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:36.464871    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8568fcf948-77q2n_b3ea20f8-efcc-4b1d-9066-e42907f636e2/kube-auth-proxy/0.log"
Sep 16 13:23:37.436878 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:37.436849    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-98h2w_f8bf39c4-ec29-4273-ab54-62522f131a6e/main/0.log"
Sep 16 13:23:37.551111 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:37.551087    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vrmzp_542b8a7d-4fe4-4818-a18c-8c8fd092f562/main/0.log"
Sep 16 13:23:38.137252 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:38.137225    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-65flk_b0a2f058-2786-422b-9410-1e0b517a366e/main/0.log"
Sep 16 13:23:47.082828 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:47.082795    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wn7dw_a0f74e27-80ff-462f-a926-9e533448687f/global-pull-secret-syncer/0.log"
Sep 16 13:23:47.222156 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:47.222123    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tjgsv_b0739542-380d-4bdd-b027-d47a25b1645e/konnectivity-agent/0.log"
Sep 16 13:23:47.311684 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:47.311660    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-72.ec2.internal_be62b2323e6968ec6b10000746a16003/haproxy/0.log"
Sep 16 13:23:51.924584 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:51.924549    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cb5d9656f-dxhbx_deb60e74-887e-46af-bc89-3216fd090dc7/authorino/0.log"
Sep 16 13:23:52.034558 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:52.034529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8vkn5_7bd517a2-175f-4f07-a62b-d5733ceca442/kuadrant-console-plugin/0.log"
Sep 16 13:23:52.120654 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:52.120621    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-wxzvf_7842f60e-a30c-497b-b11d-6841236c8dc0/limitador/0.log"
Sep 16 13:23:53.612467 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.612441    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/alertmanager/0.log"
Sep 16 13:23:53.633474 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.633443    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/config-reloader/0.log"
Sep 16 13:23:53.655661 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.655632    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/kube-rbac-proxy-web/0.log"
Sep 16 13:23:53.676020 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.675982    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/kube-rbac-proxy/0.log"
Sep 16 13:23:53.696505 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.696460    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/kube-rbac-proxy-metric/0.log"
Sep 16 13:23:53.719479 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.719454    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/prom-label-proxy/0.log"
Sep 16 13:23:53.741898 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.741871    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_92c5fe5b-b7ad-4039-8668-e74bf06d7d73/init-config-reloader/0.log"
Sep 16 13:23:53.908866 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:53.908821    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5dcb4547f6-vdppr_e54a105a-2236-4841-8143-49986fb19dfb/metrics-server/0.log"
Sep 16 13:23:54.028456 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.028423    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9dw2_664066a5-a1d8-4be9-ad02-ab9a485e70b0/node-exporter/0.log"
Sep 16 13:23:54.079779 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.079757    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9dw2_664066a5-a1d8-4be9-ad02-ab9a485e70b0/kube-rbac-proxy/0.log"
Sep 16 13:23:54.100248 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.100224    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9dw2_664066a5-a1d8-4be9-ad02-ab9a485e70b0/init-textfile/0.log"
Sep 16 13:23:54.457457 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.457416    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/prometheus/0.log"
Sep 16 13:23:54.484909 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.484863    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/config-reloader/0.log"
Sep 16 13:23:54.537629 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.537584    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/thanos-sidecar/0.log"
Sep 16 13:23:54.593040 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.593017    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/kube-rbac-proxy-web/0.log"
Sep 16 13:23:54.629361 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.629303    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/kube-rbac-proxy/0.log"
Sep 16 13:23:54.665957 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.665932    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/kube-rbac-proxy-thanos/0.log"
Sep 16 13:23:54.694072 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:54.694040    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8d230233-669d-4457-be09-6784a6a5552b/init-config-reloader/0.log"
Sep 16 13:23:56.341737 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.341697    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-2xbhj_85f9f824-9735-4103-bae9-8ba69dd5fae8/networking-console-plugin/0.log"
Sep 16 13:23:56.932501 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.932471    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"]
Sep 16 13:23:56.932826 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.932814    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452a6a0e-2afc-41bd-a708-d35f6fc82f07" containerName="maas-api"
Sep 16 13:23:56.932826 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.932827    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="452a6a0e-2afc-41bd-a708-d35f6fc82f07" containerName="maas-api"
Sep 16 13:23:56.932958 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.932893    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="452a6a0e-2afc-41bd-a708-d35f6fc82f07" containerName="maas-api"
Sep 16 13:23:56.935868 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.935852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:56.938466 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.938442    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fs84m\"/\"kube-root-ca.crt\""
Sep 16 13:23:56.939160 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.939143    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fs84m\"/\"default-dockercfg-rz5kq\""
Sep 16 13:23:56.939219 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.939162    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fs84m\"/\"openshift-service-ca.crt\""
Sep 16 13:23:56.944542 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:56.944517    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"]
Sep 16 13:23:57.015845 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.015808    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-lib-modules\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.015845 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.015848    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-proc\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.016103 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.015888    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-podres\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.016103 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.015918    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-sys\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.016103 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.016027    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlrz\" (UniqueName: \"kubernetes.io/projected/a61c85fe-75c4-4a21-a49f-68591c62e4b3-kube-api-access-xhlrz\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.116786 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.116754    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlrz\" (UniqueName: \"kubernetes.io/projected/a61c85fe-75c4-4a21-a49f-68591c62e4b3-kube-api-access-xhlrz\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.116977 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.116816    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-lib-modules\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.116977 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.116844    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-proc\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.116977 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.116883    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-podres\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.116977 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.116914    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-sys\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.117171 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.116980    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-lib-modules\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.117171 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.117002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-sys\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.117171 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.117003    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-proc\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.117171 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.117051    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a61c85fe-75c4-4a21-a49f-68591c62e4b3-podres\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.128647 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.128624    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlrz\" (UniqueName: \"kubernetes.io/projected/a61c85fe-75c4-4a21-a49f-68591c62e4b3-kube-api-access-xhlrz\") pod \"perf-node-gather-daemonset-6cz8d\" (UID: \"a61c85fe-75c4-4a21-a49f-68591c62e4b3\") " pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.245922 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.245834    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:57.523627 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.523597    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"]
Sep 16 13:23:57.525565 ip-10-0-143-72 kubenswrapper[2559]: W0916 13:23:57.525534    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda61c85fe_75c4_4a21_a49f_68591c62e4b3.slice/crio-3c454658b469895b3a8aec5e6293604c6c251d1fae74c85e8ce658a7914c36a0 WatchSource:0}: Error finding container 3c454658b469895b3a8aec5e6293604c6c251d1fae74c85e8ce658a7914c36a0: Status 404 returned error can't find the container with id 3c454658b469895b3a8aec5e6293604c6c251d1fae74c85e8ce658a7914c36a0
Sep 16 13:23:57.596462 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:57.596435    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d" event={"ID":"a61c85fe-75c4-4a21-a49f-68591c62e4b3","Type":"ContainerStarted","Data":"3c454658b469895b3a8aec5e6293604c6c251d1fae74c85e8ce658a7914c36a0"}
Sep 16 13:23:58.602306 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:58.602265    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d" event={"ID":"a61c85fe-75c4-4a21-a49f-68591c62e4b3","Type":"ContainerStarted","Data":"ef74c980818c5eeee52fa8eebacc94fb89b2893376025fec789219f7c28bee9b"}
Sep 16 13:23:58.602687 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:58.602380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:23:58.630843 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:58.630792    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d" podStartSLOduration=2.630778486 podStartE2EDuration="2.630778486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 13:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 13:23:58.629454222 +0000 UTC m=+2740.712897236" watchObservedRunningTime="2026-09-16 13:23:58.630778486 +0000 UTC m=+2740.714221499"
Sep 16 13:23:59.725722 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:59.725698    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vls4q_60881a7a-95f1-4713-bddd-7e0e36b2e403/dns/0.log"
Sep 16 13:23:59.761125 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:59.761098    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vls4q_60881a7a-95f1-4713-bddd-7e0e36b2e403/kube-rbac-proxy/0.log"
Sep 16 13:23:59.882473 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:23:59.882440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v47l6_9fb924aa-f0e1-46e9-97d6-e5db3cdd500c/dns-node-resolver/0.log"
Sep 16 13:24:00.536397 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:00.536363    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b9c897d6f-pxckm_b3aa768c-66e0-4780-8362-08249fc9f1cc/registry/0.log"
Sep 16 13:24:00.743766 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:00.743732    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jxn2l_c48de37d-b52d-44e7-bd6a-4515f7011d47/node-ca/0.log"
Sep 16 13:24:02.481233 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:02.481205    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8568fcf948-77q2n_b3ea20f8-efcc-4b1d-9066-e42907f636e2/kube-auth-proxy/0.log"
Sep 16 13:24:03.718693 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:03.718669    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zsmv_1aca5a17-6f82-43f7-9b53-c80428d48ac2/serve-healthcheck-canary/0.log"
Sep 16 13:24:04.440190 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:04.440160    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-jtt5w_0cf89e64-8aff-4bdd-a473-0e42517666ed/insights-operator/0.log"
Sep 16 13:24:04.441560 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:04.441541    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-jtt5w_0cf89e64-8aff-4bdd-a473-0e42517666ed/insights-operator/1.log"
Sep 16 13:24:04.472701 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:04.472673    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8lq9_f883c3a1-fb07-4fbb-998a-4fea855834c0/kube-rbac-proxy/0.log"
Sep 16 13:24:04.535335 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:04.535306    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8lq9_f883c3a1-fb07-4fbb-998a-4fea855834c0/exporter/0.log"
Sep 16 13:24:04.605547 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:04.605517    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8lq9_f883c3a1-fb07-4fbb-998a-4fea855834c0/extractor/0.log"
Sep 16 13:24:04.616871 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:04.616846    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fs84m/perf-node-gather-daemonset-6cz8d"
Sep 16 13:24:08.438665 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:08.438636    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8568fcf948-77q2n_b3ea20f8-efcc-4b1d-9066-e42907f636e2/kube-auth-proxy/0.log"
Sep 16 13:24:08.468072 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:08.468008    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-k7b8l_e746e6c0-4127-42e7-b184-d8e90274bf4d/manager/0.log"
Sep 16 13:24:08.522746 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:08.522702    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c7bc95455-k7b8l_e746e6c0-4127-42e7-b184-d8e90274bf4d/copy-manifests/0.log"
Sep 16 13:24:09.915907 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:09.915879    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-575fbb86f4-cndrn_e3e00e7b-dab6-4162-ab38-ffd546518889/manager/0.log"
Sep 16 13:24:10.788226 ip-10-0-143-72 kubenswrapper[2559]: I0916 13:24:10.788158    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8568fcf948-77q2n_b3ea20f8-efcc-4b1d-9066-e42907f636e2/kube-auth-proxy/0.log"